o W hv @s,dZddlZddlZddlZddlZddlZddlZddlmZddl Z ddl Z ddl Z ddlmZmZddlmZmZddlmZddlmZmZmZmZmZmZmZddlmZm Z m!Z!m"Z"m#Z#dd l$m%Z%dd l&m'Z'dd l(m)Z)m*Z*ed dd ddddddf ddZ+GdddZ,GdddZ-dS)a Benchmark a YOLO model formats for speed and accuracy. Usage: from ultralytics.utils.benchmarks import ProfileModels, benchmark ProfileModels(['yolo11n.yaml', 'yolov8s.yaml']).profile() benchmark(model='yolo11n.pt', imgsz=160) Format | `format=argument` | Model --- | --- | --- PyTorch | - | yolo11n.pt TorchScript | `torchscript` | yolo11n.torchscript ONNX | `onnx` | yolo11n.onnx OpenVINO | `openvino` | yolo11n_openvino_model/ TensorRT | `engine` | yolo11n.engine CoreML | `coreml` | yolo11n.mlpackage TensorFlow SavedModel | `saved_model` | yolo11n_saved_model/ TensorFlow GraphDef | `pb` | yolo11n.pb TensorFlow Lite | `tflite` | yolo11n.tflite TensorFlow Edge TPU | `edgetpu` | yolo11n_edgetpu.tflite TensorFlow.js | `tfjs` | yolo11n_web_model/ PaddlePaddle | `paddle` | yolo11n_paddle_model/ MNN | `mnn` | yolo11n.mnn NCNN | `ncnn` | yolo11n_ncnn_model/ RKNN | `rknn` | yolo11n_rknn_model/ NPath)YOLO YOLOWorld) TASK2DATA TASK2METRIC)export_formats)ARM64ASSETSLINUXLOGGERMACOSTQDM WEIGHTS_DIR)IS_PYTHON_3_13 check_imgszcheck_requirements check_yolo is_rockchip) safe_download) file_size) get_cpu_info select_devicez yolo11n.ptFcpuMbP?c #st|}t|tr|d|dksJd ddldjj_djj_t|dd }t|t t fr5t |}t |j j d d d} |pEt|j}t|j} g} t} |} | rmttd }||vsmJd |d| dtttD]\}\}}}}}}d\}}z| r| |krWqv|dkr|jdksJdn|dkrtrtrJdn|dvrtstrtrJd|dvrtrJd|dvrt|trJd|dkrt|trJd|jdksJd| rJdtstsJd |d!krt|trJd"|d#krt|trJd$|d%kr:| rJt|tr%Jd&|jd'ks/Jd(d)|vs:Jd*|d+kr_t|trIJd,| rPJd-tsWJd.tr_Jd/d0|jvrl|slJd1d2|jvry|syJd3|d4kr|j p|j!p|j"}|}n|j#||||||dd5}t ||jd6}|t |vsJd7d8}|jd9ks|dksJd:|d;vsJd<|d=kst$%d>ksJd?|d@vr| rJdA|j&t'dB|||ddC|j(|d|d|||ddD}|j)| |j*dE}}t+dF||dG}| ,|dHt+t-|dt+|dIt+|dG|gWqvt.yf}z5|rAt|t/usAJdJ|dK|t01dJ|dK|| ,||t+t-|ddddgWYd}~qvd}~wwt2|dLj3| dMdNdO| dPdQgdR}|j"}t| }dS}dT|dU|dV|dW|dXdY|dZ|4d4dZ } t05| t6d[d\d]d^d_ }!|!7| Wdn 1swY|rt|t8r|| j9}"|t:fd`da|"DsJdb|S)caU Benchmark a YOLO model across different formats for speed and accuracy. Args: model (str | Path): Path to the model file or directory. data (str | None): Dataset to evaluate on, inherited from TASK2DATA if not passed. imgsz (int): Image size for the benchmark. half (bool): Use half-precision for the model if True. int8 (bool): Use int8-precision for the model if True. device (str): Device to run the benchmark on, either 'cpu' or 'cuda'. verbose (bool | float): If True or a float, assert benchmarks pass with given metric. eps (float): Epsilon value for divide by zero prevention. format (str): Export format for benchmarking. If not supplied all formats are benchmarked. Returns: (pandas.DataFrame): A pandas DataFrame with benchmark results for each format, including file size, metric, and inference time. Examples: Benchmark a YOLO model with default settings: >>> from ultralytics.utils.benchmarks import benchmark >>> benchmark(model="yolo11n.pt", imgsz=640) rTz'benchmark() only supports square imgsz.N xF)verboseend2endArgumentzExpected format to be one of z , but got 'z'.)❌Nobbz.TensorFlow GraphDef not supported for OBB task z3Edge TPU export only supported on non-aarch64 Linux>rzECoreML and TF.js export only supported on macOS and non-aarch64 Linux>r(z#CoreML not supported on Python 3.13>r%r'rz;YOLOWorldv2 TensorFlow exports not supported by onnx2tf yet z,YOLOWorldv2 Paddle exports not supported yetzBPaddle OBB bug https://github.com/PaddlePaddle/Paddle/issues/72024z3End-to-end models not supported by PaddlePaddle yetz(Windows Paddle exports not supported yet z)YOLOWorldv2 MNN exports not supported yet z*YOLOWorldv2 NCNN exports not supported yetz%YOLOWorldv2 IMX exports not supporteddetectz%IMX only supported for detection taskC2fzIMX only supported for YOLOv8z*YOLOWorldv2 RKNN exports not supported yetz+End-to-end models not supported by RKNN yetzRKNN only supported on Linuxz1RKNN Inference only supported on Rockchip devicesrzinference not supported on CPUcudazinference not supported on GPU-)imgszformathalfint8datadevicer )taskz export failedu❎posez(GraphDef Pose inference is not supported>r'rzinference not supportedr(Darwinz(inference only supported on macOS>=10.13>r-zHEnd-to-end torch.topk operation is not supported for NCNN prediction yetzbus.jpg)r4r9r6r )r8batchr4plotsr9r6r7r inferenceu✅zBenchmark failure for : )r9Formatu Status❔z Size (MB)zInference time (ms/im)FPS)columnsudBenchmarks legend: - ✅ Success - ❎ Export passed but validation failed - ❌️ Export failedz Benchmarks complete for z on z at imgsz=z (z.2fzs)  zbenchmarks.logaignoreutf-8)errorsencodingc3s"|] }|r|kVqdSN)notna).0xfloorpdP/var/www/vscode/kcb/lib/python3.10/site-packages/ultralytics/utils/benchmarks.py  zbenchmark..z%Benchmark failure: metric(s) < floor );r isinstancelistpandasoptionsdisplay max_columnswidthrstrrrgetattrmodelrr:rtimelower frozensetr enumeratezipvaluesr r r rr__str__rtypept_path ckpt_path model_nameexportplatformsystempredictr val results_dictspeedroundappendr ExceptionAssertionErrorr errorr DataFramefillnainfoopenwritefloatarrayall)#rar8r4r6r7r9r epsr5 is_end2endkeyyt0 format_argformatsinamesuffixrgpu_emojifilenameexported_modelresultsmetricrsfpsedfdtlegendsfmetricsrTrQrU benchmark3s"$    (             0",  2   (rc@s>eZdZdZddZddZdddZed d Zd d Z d S)RF100Benchmarka Benchmark YOLO model performance across various formats for speed and accuracy. This class provides functionality to benchmark YOLO models on the RF100 dataset collection. Attributes: ds_names (List[str]): Names of datasets used for benchmarking. ds_cfg_list (List[Path]): List of paths to dataset configuration files. rf (Roboflow): Roboflow instance for accessing datasets. val_metrics (List[str]): Metrics used for validation. Methods: set_key: Set Roboflow API key for accessing datasets. parse_dataset: Parse dataset links and download datasets. fix_yaml: Fix train and validation paths in YAML files. evaluate: Evaluate model performance on validation results. cCs g|_g|_d|_gd|_dS)zcInitialize the RF100Benchmark class for benchmarking YOLO model performance across various formats.Nclassimagestargets precisionrecallmap50map95)ds_names ds_cfg_listrf val_metrics)selfrTrTrU__init__szRF100Benchmark.__init__cCs$tdddlm}||d|_dS)a% Set Roboflow API key for processing. Args: api_key (str): The API key. Examples: Set the Roboflow API key for accessing datasets: >>> benchmark = RF100Benchmark() >>> benchmark.set_key("your_roboflow_api_key") roboflowr)Roboflow)api_keyN)rrrr)rrrrTrTrUset_keys zRF100Benchmark.set_keydatasets_links.txtc Cs tjdrtdtdfntdtdtdtdt|ddY}|D]N}zCt d| \}}}}}|j ||d|} t| s_|j|||dntd |j t| d Wq,tyzYq,wWd n1swY|j |jfS) a Parse dataset links and download datasets. Args: ds_link_txt (str): Path to the file containing dataset links. Returns: ds_names (List[str]): List of dataset names. ds_cfg_list (List[Path]): List of paths to dataset configuration files. Examples: >>> benchmark = RF100Benchmark() >>> benchmark.set_key("api_key") >>> benchmark.parse_dataset("datasets_links.txt") zrf-100zultralytics-benchmarkszQhttps://github.com/ultralytics/assets/releases/download/v0.0.0/datasets_links.txtrJrLz/+r3yolov8zDataset already downloaded.z data.yamlN)ospathexistsshutilrmtreemkdirchdirrr|resplitstriprrurr workspaceprojectversiondownloadr r{rcwdrv) r ds_link_txtfilelinerurlrrr proj_versionrTrTrU parse_datasets*(        zRF100Benchmark.parse_datasetcCst|dd }t|}Wdn1swYd|d<d|d<t|ddd}t||WddS1sq0dttd d d d D| fd dDq0Wdn1skwYd} t | dkrt d| D] } | ddkr| d} qnt ddd | Dd} t|ddd}| |j |d| dWddS1swYdS)a Evaluate model performance on validation results. Args: yaml_path (str): Path to the YAML configuration file. val_log_file (str): Path to the validation log file. eval_log_file (str): Path to the evaluation log file. list_ind (int): Index of the current dataset in the list. Returns: (float): The mean average precision (mAP) value for the evaluated model. Examples: Evaluate a model on a specific dataset >>> benchmark = RF100Benchmark() >>> benchmark.evaluate("path/to/data.yaml", "path/to/val_log.txt", "path/to/eval_log.txt", 0) )u🚀u⚠️u💡r$rJrnamesNc3s|]}|vVqdSrMrT)rOsymbol)rrTrUrV>sz*RF100Benchmark.evaluate.. cSs|dkS)NrrT)rqrTrTrUAsz)RF100Benchmark.evaluate..cSsg|]}|dqS)rG)rrOrrTrTrU Bsz+RF100Benchmark.evaluate..c 3s`|]+}|vs|dkrdvrdvrddddddd d Vqd S) rz(AP)z(AR)rrrArBr(r)rNrTr) class_namesentriesrTrUrVCs  rzMultiple dicts foundrrrzSingle dict foundcSsg|]}|dqS)rrT)rOresrTrTrUrXrrHrCrG)r|rr readlinesanyrrYfilterextendlenr r{r}r) r yaml_path val_log_file eval_log_filelist_ind skip_symbolsstreamrlines eval_linesmap_vallstrT)rrrrUevaluate%s>      "zRF100Benchmark.evaluateN)r) __name__ __module__ __qualname____doc__rrr staticmethodrrrTrTrTrUrs %  rc@seZdZdZ       d#defd d Zd d Zd dZede fddZ ed$ddZ d%de de fddZ d%de de fddZddZedd Zed!d"ZdS)& ProfileModelsa( ProfileModels class for profiling different models on ONNX and TensorRT. This class profiles the performance of different models, returning results such as model speed and FLOPs. Attributes: paths (List[str]): Paths of the models to profile. num_timed_runs (int): Number of timed runs for the profiling. num_warmup_runs (int): Number of warmup runs before profiling. min_time (float): Minimum number of seconds to profile for. imgsz (int): Image size used in the models. half (bool): Flag to indicate whether to use FP16 half-precision for TensorRT profiling. trt (bool): Flag to indicate whether to profile using TensorRT. device (torch.device): Device used for profiling. Methods: profile: Profiles the models and prints the result. get_files: Gets all relevant model files. get_onnx_model_info: Extracts metadata from an ONNX model. iterative_sigma_clipping: Applies sigma clipping to remove outliers. profile_tensorrt_model: Profiles a TensorRT model. profile_onnx_model: Profiles an ONNX model. generate_table_row: Generates a table row with model metrics. generate_results_dict: Generates a dictionary of profiling results. print_table: Prints a formatted table of results. Examples: Profile models and print results >>> from ultralytics.utils.benchmarks import ProfileModels >>> profiler = ProfileModels(["yolo11n.yaml", "yolov8s.yaml"], imgsz=640) >>> profiler.profile() dr<TNpathsc CsL||_||_||_||_||_||_||_|p"ttj r dnd|_dS)a[ Initialize the ProfileModels class for profiling models. Args: paths (List[str]): List of paths of the models to be profiled. num_timed_runs (int): Number of timed runs for the profiling. num_warmup_runs (int): Number of warmup runs before the actual profiling starts. min_time (float): Minimum time in seconds for profiling a model. imgsz (int): Size of the image used during profiling. half (bool): Flag to indicate whether to use FP16 half-precision for TensorRT profiling. trt (bool): Flag to indicate whether to profile using TensorRT. device (torch.device | None): Device used for profiling. If None, it is determined automatically. Notes: FP16 'half' argument option removed for ONNX as slower on CPU than FP32. Examples: Initialize and profile models >>> from ultralytics.utils.benchmarks import ProfileModels >>> profiler = ProfileModels(["yolo11n.yaml", "yolov8s.yaml"], imgsz=640) >>> profiler.profile() rrN) rnum_timed_runsnum_warmup_runsmin_timer4r6trttorchr9r2 is_available) rrrrrr4r6rr9rTrTrUrs!"zProfileModels.__init__c Cs|}|s tddSg}g}|D]t}|d}|jdvrStt|}||}|j rG|j j dkrG| sG|j d|j|j|j dd}|j d |j|j dd }n|jd kr`||}|}nq|t|} |t|} |||j| | ||||j| | |q|||S) a Profile YOLO models for speed and accuracy across various formats including ONNX and TensorRT. Returns: (List[Dict]): List of dictionaries containing profiling results for each model. Examples: Profile models and print results >>> from ultralytics.utils.benchmarks import ProfileModels >>> profiler = ProfileModels(["yolo11n.yaml", "yolov8s.yaml"]) >>> results = profiler.profile() z'No matching *.pt or *.onnx files found.Nz.engine.pt.yml.yamlrengineF)r5r6r4r9r onnx)r5r4r9r z.onnx) get_filesr warning with_suffixrrr_fuser{rr9riis_filermr6r4get_onnx_model_infoprofile_tensorrt_modelprofile_onnx_modelrugenerate_table_rowstemgenerate_results_dict print_table) rfiles table_rowsoutputr engine_filera model_info onnx_filet_enginet_onnxrTrTrUprofilesH        zProfileModels.profilecsg}|jD]2tr gd}|fdd|Dqjdvr-|tq|ttqt dt |ddt |DS)z Return a list of paths for all relevant model files given by the user. Returns: (List[Path]): List of Path objects for the model files. )z*.ptz*.onnxz*.yamlcs(g|]}tt|D]}|q qSrT)globr_)rOextrrrTrUrs(z+ProfileModels.get_files..rz Profiling: cSsg|]}t|qSrTr)rOrrTrTrUrr) rris_dirrrrur_rr r{sorted)rr  extensionsrTrrUrs  zProfileModels.get_filesrcCsdS)zXExtracts metadata from an ONNX model file including parameters, GFLOPs, and input shape.)rrrrrT)rrTrTrUrsz!ProfileModels.get_onnx_model_inforArcCspt|}t|D],}t|t|}}|||||k||||k@}t|t|kr3|S|}q |S)a Apply iterative sigma clipping to data to remove outliers. Args: data (numpy.ndarray): Input data array. sigma (float): Number of standard deviations to use for clipping. max_iters (int): Maximum number of iterations for the clipping process. Returns: (numpy.ndarray): Clipped data array with outliers removed. )nprrangemeanstdr)r8sigma max_itersrrr clipped_datarTrTrUiterative_sigma_clippings $z&ProfileModels.iterative_sigma_clippingrrrc Cs |jr t|s dSt|}tj|j|jdftjd}d}tdD]}t }t|j D] }|||jddq-t |}q"t t |j |||j |jd}g} tt||dD]}|||jdd} | | d jd q[|jt| d dd } t| t| fS) a Profile YOLO model performance with TensorRT, measuring average run time and standard deviation. Args: engine_file (str): Path to the TensorRT engine file. eps (float): Small epsilon value to prevent division by zero. Returns: mean_time (float): Mean inference time in milliseconds. std_time (float): Standard deviation of inference time in milliseconds. )rrr)dtyperF)r4r 2descrr?rArr)rrrrrzerosr4uint8rrbrmaxrtrrrrursr!rrr) rrrra input_dataelapsedr start_timenum_runs run_timesrrTrTrUrs"  $z$ProfileModels.profile_tensorrt_modelcCstdddl}|}|jj|_d|_|j||dgd}|d}|j }t dd|j D }|r;d d |j |j fn|j } d |vrFt j} n'd |vrNt j} nd |vrVt j} nd|vr^t j} nd|vrft j} ntd|t jj| | } |j} |dj} d}td D]}t}t|jD] }|| g| | iqt|}qtt|j|||j|j }g}t!t||dD]}t}|| g| | i|"t|dq|j#t $|ddd}t %|t &|fS)a Profile an ONNX model, measuring average inference time and standard deviation across multiple runs. Args: onnx_file (str): Path to the ONNX model file. eps (float): Small epsilon value to prevent division by zero. Returns: mean_time (float): Mean inference time in milliseconds. std_time (float): Standard deviation of inference time in milliseconds. onnxruntimerNr*CPUExecutionProvider) providerscss"|] }t|to |dkVqdS)rN)rXint)rOdimrTrTrUrVOrWz3ProfileModels.profile_onnx_model..rrfloat16r~doubleint64int32zUnsupported ONNX datatype rr$r@rAr(r&)'rr/SessionOptionsGraphOptimizationLevelORT_ENABLE_ALLgraph_optimization_levelintra_op_num_threadsInferenceSession get_inputsrirshaper4rr4float32float64r6r7 ValueErrorrandomrandastyper get_outputsrrbrrunr)rtrrrrur!rrr)rrrort sess_optionssess input_tensor input_typedynamic input_shape input_dtyper* input_name output_namer+rr,r-r.rTrTrUr8sL     z ProfileModels.profile_onnx_modelc Csf|\}}}}d|dd|jd|ddd|ddd |ddd|ddd |d dd|dd S) a Generate a table row string with model performance metrics. Args: model_name (str): Name of the model. t_onnx (tuple): ONNX model inference time statistics (mean, std). t_engine (tuple): TensorRT engine inference time statistics (mean, std). model_info (tuple): Model information (layers, params, gradients, flops). Returns: (str): Formatted table row string with model metrics. z| 18sz | z | - | rz.1f±rz ms | g.Az |)r4) rrlrrrlayersparams gradientsflopsrTrTrUrys 6z ProfileModels.generate_table_rowcCs6|\}}}}||t|dt|ddt|dddS)a Generate a dictionary of profiling results. Args: model_name (str): Name of the model. t_onnx (tuple): ONNX model inference time statistics (mean, std). t_engine (tuple): TensorRT engine inference time statistics (mean, std). model_info (tuple): Model information (layers, params, gradients, flops). Returns: (dict): Dictionary containing profiling results. rr)z model/namezmodel/parametersz model/GFLOPszmodel/speed_ONNX(ms)zmodel/speed_TensorRT(ms))rt)rlrrrrTrUrVrWrTrTrUr s   z#ProfileModels.generate_results_dictcCstjr tjdnd}ddddtdd|d d d g}d d d d|Dd }d d dd|Dd }td|t||D]}t|qJdS)z Print a formatted table of model profiling results. Args: table_rows (List[str]): List of formatted table row strings. rGPUModelzsize
(pixels)zmAPval
50-95zSpeed
CPU (z) ONNX
(ms)zSpeed
z TensorRT
(ms)zparams
(M)zFLOPs
(B)|css|] }d|dVqdS)rNrTrOhrTrTrUrVsz,ProfileModels.print_table..css |] }dt|dVqdS)r3rAN)rr[rTrTrUrVsz N)rr2rget_device_namerjoinr r{)r rheadersheader separatorrowrTrTrUr s      zProfileModels.print_table)rrrrTTN)rAr)r)rrrrrYrrrrr_rr!r~rrrr r rTrTrTrUr^s2$ *7 'A r).rrrrnrrrbpathlibrnumpyr torch.cudarr ultralyticsrrultralytics.cfgrrultralytics.engine.exporterrultralytics.utilsr r r r r rrultralytics.utils.checksrrrrrultralytics.utils.downloadsrultralytics.utils.filesrultralytics.utils.torch_utilsrrrrrrTrTrTrUsB  $