o V hʼn@sddlZddlZddlZddlZddlZddlZddlZddlmZddl m Z ddl m Z ddl Z ddlZddlmZmZddlmZddlmZmZmZmZmZmZmZmZmZmZm Z m!Z!ddl"m#Z#m$Z$m%Z%dd l&m'Z'm(Z(m)Z)dd l*m+Z+d Z,hd Z-hd Z.e/e0dd1dkZ2de-de.Z3ddZ4d>ddZ5ddZ6dejfddZ7d d!Z8d"d#Z9d$d%Z:d?d'd(Z;d@d)d*ZdAd1d2Z?dBd3d4Z@Gd5d6d6ZAdCd8d9ZBd:d;ZCd bmpdngjpgmpopfmpngtifheicjpegtiffwebp> tsasfavigifm4vmkvmovmp4mpgwmvmpegwebm PIN_MEMORYTtruezSupported formats are: images: z videos: cs:tjdtjtjdtjfdd|DS)z0Define label paths as a function of image paths.imageslabelscs.g|]}|dddddqS).rz.txt)joinrsplit.0xsasbJ/var/www/vscode/kcb/lib/python3.10/site-packages/ultralytics/data/utils.py /s.z#img2label_paths..)ossep) img_pathsr@r=rAimg2label_paths,s&rF 2c Cs|rt|dkrt|ddSt|t|t|}g}g}g}|D]W}zLt} t |j } | t| d| | t} t |d } | } Wdn1sZwYt| } | dkrr| | d| Wq%ty|Yq%w|st|ddSt|}t|dkrtj|dd nd}d t|d d d }d|d d|d d}|rt|}t|dkrtj|dd nd}d|d d|d d}nd}||ks||krt|d|||ddSt|d|||ddS)am Check dataset file access speed and provide performance feedback. This function tests the access speed of dataset files by measuring ping (stat call) time and read speed. It samples up to 5 files from the provided list and warns if access times exceed the threshold. Args: files (list): List of file paths to check for access speed. threshold_ms (float, optional): Threshold in milliseconds for ping time warnings. threshold_mb (float, optional): Threshold in megabytes per second for read speed warnings. max_files (int, optional): The maximum number of files to check. prefix (str, optional): Prefix string to add to log messages. Examples: >>> from pathlib import Path >>> image_files = list(Path("dataset/images").glob("*.jpg")) >>> check_file_speeds(image_files, threshold_ms=15) rz%Image speed checks: No files to checkNirbiz*Image speed checks: failed to access filesr6)ddofz, size: i.1fz KBzping: ±z msz, read: z MB/srJuFast image access ✅ ()zSlow image access detected (z). Use local storage instead of remote/mounted storage for better performance. See https://docs.ultralytics.com/guides/model-training-tips/)lenr warningrandomsamplemintime perf_counterrCstatst_sizeappendopenread Exceptionnpmeanstdinfo)files threshold_ms threshold_mb max_filesprefix ping_times file_sizes read_speedsfstart file_sizefile_obj_ read_timeavg_pingstd_pingsize_msgping_msg avg_speed std_speed speed_msgr@r@rAcheck_file_speeds2sR        "rvc Csbd}|D]}z |t|j7}WqtyYqwtt|}|d || S)z?Returns a single hash value of a list of paths (files or dirs).rrJ) rCrWrXOSErrorhashlibsha256strencodeupdater8 hexdigest)pathssizephr@r@rAget_hash~s rimgcCsl|j}|jdkr4z!|}r$|dd}|dvr'|d|df}W|SW|SW|Sty3Y|Sw|S)z Returns exif-corrected PIL size.JPEGiN>r6r)rformatgetexifgetr\)rsexifrotationr@r@rA exif_sizes"    rc Cs\|\\}}}d\}}}z}t|}|t|}|d|df}|ddk|ddk@s5Jd|d|jtvsGJd|jdt|jd vrt|d *} | d d | d krvt t|j |dddd||d}Wdn1swYd}Wnt y} zd}||d| }WYd} ~ nd} ~ ww||f|||fS)zVerify one image.)rrrJr6r image size <10 pixelszInvalid image format . rr#rKrd subsamplingquality!: corrupt JPEG restored and savedN : ignoring corrupt image/label: )rrZverifyrrlower IMG_FORMATSFORMATS_HELP_MSGseekr[rexif_transposesaver\) argsim_fileclsrenfncmsgimshaperier@r@rA verify_images0   ($   rc s|\}}}}}}}}dddddgdf\} } } } } }z(t|}|t|}|d|df}|ddk|ddk@sEJd|d|jtvsWJd|jd t|jd vrt|d *}|d d | dkrt t|j |dddd||d} Wdn1swYt j|rd} t|ddJ}dd| D}tdd|Dr|stjdd|Dtjd}dd|Dt|ddtfd}tj|tjd}Wdn1swYt|}r|r/|jdd||ksJdd||d|ddddfd|dddd f}n|jddksBJd |jdd!|ddddf}|dks^Jd"||dk|dkspJd#||dk|r{d|dddf<|dddf}||ksJd$t|d%|d&|dtj|dd'd(\}}t||kr||}rfd)d|D||d*|t|d+} n+d} tjd|rd||ndftjd}nd} tjd|rd||ndftjd}|r-|ddddfd||}|d kr-t |d,dk|d-dkBd.d/!tj}tj||d0gdd1}|ddddf}||||| | | | | f WSt"ym}zd} ||d2|} ddddd| | | | | g WYd}~Sd}~ww)3zVerify one image-label pair.rrJNr6rrrzinvalid image format rrrKrrrrrrrutf-8encodingcSsg|] }t|r|qSr@)rPsplitr:r@r@rArBz&verify_image_label..css|] }t|dkVqdS)rN)rPr:r@r@rA sz%verify_image_label..cSsg|]}|dqS)rr@r:r@r@rArBdtypecSs,g|]}tj|ddtjdddqS)r6Nrr)r]arrayfloat32reshaper:r@r@rArB,rrIzlabels require z columns eachzlabels require 5 columns, z columns detectedz,non-normalized or out of bounds coordinates znegative label values z Label class z exceeds dataset class count z. Possible class labels are 0-T)axis return_indexcsg|]}|qSr@r@r:segmentsr@rArBr: z duplicate labels removed).r).r6g?).N)rr)#rrZrrrrrrrr[rrrrCpathisfilestrip splitlinesanyr]rr concatenaterrrPrmaxrTintuniquezeroswhereastyper\)rrlb_filerekeypointnum_clsnkptndim single_clsnmrnerr keypointsrrrilbclassesnlpointsmax_clsrmikpt_maskrr@rrAverify_image_labels  ($   0.&$$ (& *$rc Csddlm}ddlm}tt|}|jdd\}}g}t|dd=} | D]2} t t | \} } } }}| |d|}| |d|}||}||}| ||||t | fq(Wdn1sewY|d\}}|D]K\}}}}}tdd ||d D}|j||f||d|d d }||d |dd|dd|d}|j||d|||dkrdnd|dqs|||dS)a Visualizes YOLO annotations (bounding boxes and class labels) on an image. This function reads an image and its corresponding annotation file in YOLO format, then draws bounding boxes around detected objects and labels them with their respective class names. The bounding box colors are assigned based on the class ID, and the text color is dynamically adjusted for readability, depending on the background color's luminance. Args: image_path (str): The path to the image file to annotate, and it can be in formats supported by PIL. txt_path (str): The path to the annotation file in YOLO format, that should contain one line per object. label_map (dict): A dictionary that maps class IDs (integers) to class labels (strings). Examples: >>> label_map = {0: "cat", 1: "dog", 2: "bird"} # It should include all annotated classes details >>> visualize_image_annotations("path/to/image.jpg", "path/to/annotations.txt", label_map) rN)colorsrrrr6css|]}|dVqdS)Nr@)r;cr@r@rArz.visualize_image_annotations..Tnone) linewidth edgecolor facecolorgz6?g,C?g]m{?rIg?whiteblack)colorbackgroundcolor)matplotlib.pyplotpyplotultralytics.utils.plottingrr]rrrZrmapfloatrrYrsubplotstuple Rectangle add_patchtextimshowshow) image_pathtxt_path label_mappltrr img_height img_width annotationsfilelineclass_idx_centery_centerwidthheightr<ywrfigaxlabelrrect luminancer@r@rAvisualize_image_annotationss0   $*  r r6cCsptj|tjd}tj|tjd}||jdddf}tj|||d|d||d|}}t |||fS)a Convert a list of polygons to a binary mask of the specified image size. Args: imgsz (tuple): The size of the image as (height, width). polygons (list[np.ndarray]): A list of polygons. Each polygon is an array with shape [N, M], where N is the number of polygons, and M is the number of points such that M % 2 = 0. color (int, optional): The color value to fill in the polygons on the mask. downsample_ratio (int, optional): Factor by which to downsample the mask. Returns: (np.ndarray): A binary mask of the specified image size with the polygons filled in. rrrr)rr6) r]ruint8asarrayint32rrcv2fillPolyresize)imgszpolygonsrdownsample_ratiomasknhnwr@r@rA polygon2mask's rcstfdd|DS)a Convert a list of polygons to a set of binary masks of the specified image size. Args: imgsz (tuple): The size of the image as (height, width). polygons (list[np.ndarray]): A list of polygons. Each polygon is an array with shape [N, M], where N is the number of polygons, and M is the number of points such that M % 2 = 0. color (int): The color value to fill in the polygons on the masks. downsample_ratio (int, optional): Factor by which to downsample each mask. Returns: (np.ndarray): A set of binary masks of the specified image size with the polygons filled in. cs"g|] }t|dgqS)r)rrr:rrrr@rArBL"z"polygons2masks..)r]r)rrrrr@rrApolygons2masks>src Cstj|d||d|ft|dkrtjntjd}g}g}tt|D] }t|||dg|dd}|| |j || q%t |}t | }t||}tt|D]} || | d}||}tj|d| dd}q^||fS)z!Return a (640, 640) overlap mask.rr6rrr)rr)a_mina_max)r]rrPrr rangerrrYrrsumr argsortrclip) rrrmasksareasmssirindexrr@r@rApolygons2masks_overlapOs$  r'rreturncstdp td}|sJddt|dkr)fdd|D}t|dks@Jddt|d ||d S) a Find and return the YAML file associated with a Detect, Segment or Pose dataset. This function searches for a YAML file at the root level of the provided directory first, and if not found, it performs a recursive search. It prefers YAML files that have the same stem as the provided path. Args: path (Path): The directory path to search for the YAML file. Returns: (Path): The path of the found YAML file. z*.yamlzNo YAML file found in ''r6csg|] }|jjkr|qSr@)stemr;rirr@rArBurz%find_dataset_yaml..zExpected 1 YAML file in 'z ', but found z. r)listglobrglobresolverP)rrar@r,rAfind_dataset_yamles   .r1cst|}d}t|st|r#t|tddd}tt|}|jd}}t|dddD]&}|vrQ|dks9dvrEt t |d |d t d  dd<q+d vrcd vrct t |dd vrd vrtd d krt t |dtd dd dd vrddtd Dd <ntd d <td d <ddd<t|pĈdptddjstd<dD]E}|rt|tr|}|s|dr|dd}t||<qևfdd|D|<qևfdddD\}}|rddt|tr5|n|gD}td d|Dst|} t dd!| d"d#d|Dd$d%} |ri|rit | n| d&td'td%7} t| t} d} |d(r| d)rt|tdd*n|d+rt d,|d-t!"|} nt#|d.id/t$t| d0d1} | d2vrd3| d4t%d5tnd6| d7}t d8|d9t&t'd rd:Sd;S).channelsr yaml_file)r7r8testminivalz../Ncsg|] }t|qSr@)rzr0r:r,r@rArBrc3s|]}|VqdSN)rr:)datar@rArsz$check_det_dataset..)r8rcSsg|]}t|qSr@)rr0r:r@r@rArBcss|]}|VqdSrBexistsr:r@r@rArrz Dataset 'z"' images not found, missing path 'cSsg|]}|s|qSr@rEr:r@r@rArBrDrr)z% Note dataset download directory is 'z'. You can update this in 'http.zip)urlr3r5bash zRunning z ...yaml(r6zs)>Nru success ✅ z , saved to boldzfailure u ❌zDataset download  z Arial.ttfzArial.Unicode.ttf)(rzipfile is_zipfilerrrr1parentr SyntaxErrorrr rQpoprPrrrr is_absoluter0 isinstancerzrF startswithr-allrr`r FileNotFoundErrorrUendswithrCsystemexecroundrrr)dataset autodownloadr extract_dirnew_dirkr<r8rnamemtrdtr@)rCrrAcheck_det_datasetzs    $*$         ,rgc Csnt|drt|tddd}nt|jdvr#t|}t|tddd}t|}|r-|nt|}|s~t dt d|dt }t|d kr]t jd td ddd n d |d}t||jdt dt |ddtd|d|d}|st d|t|dt|d}|rddlm}t dt|d||dd}|d}n t d|d |d!r|d!n |d"r|d"nd#} |d$r|d$nd#} |d!kr| st d%| } n|d$kr| st d&| } td'd(|dd)D} d*d(|dD} ttt| } || | d+D]\} }t| d,d-|d.}|d#urHt |q+d/d(|d0D}t|}td1d2|D}|dkr| dkrst |d3| d4t |d5|d6|d7q+|| krt |d5|d6|d8| d9|d: q+t |d5|d6|d;q+|| | | | da Checks a classification dataset such as Imagenet. This function accepts a `dataset` name and attempts to retrieve the corresponding dataset information. If the dataset is not found locally, it attempts to download the dataset from the internet and save it locally. Args: dataset (str | Path): The name of the dataset. split (str, optional): The split of the dataset. Either 'val', 'test', or ''. Returns: (dict): A dictionary containing the following keys: - 'train' (Path): The directory path containing the training set of the dataset. - 'val' (Path): The directory path containing the validation set of the dataset. - 'test' (Path): The directory path containing the test set of the dataset. - 'nc' (int): The number of classes in the dataset. - 'names' (dict): A dictionary of class names in the dataset. )zhttp:/zhttps:/TFr2>.gz.tarrHrJz Dataset not found, missing path z, attempting download...imagenetrJzdata/scripts/get_imagenet.sh)shellcheckz?https://github.com/ultralytics/assets/releases/download/v0.0.0/rH)r3uDataset download success ✅ (rMz s), saved to rMrNr7z#Dataset 'split=train' not found at z*.jpgz*.pngr)split_classify_datasetzFound z1 images in subdirectories. Attempting to split...g?) train_ratiozNo images found in z or its subdirectories.r8r9Nr@z:Dataset 'split=val' not found, using 'split=test' instead.z:Dataset 'split=test' not found, using 'split=val' instead.cSsg|]}|r|qSr@)is_dirr:r@r@rArBrDz%check_cls_dataset..*cSsg|] }|r|jqSr@)rorbr:r@r@rArBsr7r8r@: ...cS&g|]}|jddtvr|qSr6Nsuffixrr)r;rr@r@rArB%&*.*cSsh|]}|jqSr@)rQ)r;rr@r@rA 'sz$check_cls_dataset..r:z:' no training images foundz found z images in z classes (no images found)z classes (requires z classes, not rOu classes ✅ r>)r7r8r@rr;r=)!rzrVrrrrxrror0r r`rQrU subprocessrunr rrQrr-r/ultralytics.data.splitrmrPerrorrFr.iterdirdict enumeratesorteditemsrX)r]rrdata_dirrdrI train_set image_filesrmval_settest_setrr;ravrerarndr@r@rAcheck_cls_datasetsr   (              , rc@s@eZdZdZdddZeddZd d Zdd d Zd dZ dS)HUBDatasetStatsa A class for generating HUB dataset JSON and `-hub` dataset directory. Args: path (str): Path to data.yaml or data.zip (with data.yaml inside data.zip). Default is 'coco8.yaml'. task (str): Dataset task. Options are 'detect', 'segment', 'pose', 'classify'. Default is 'detect'. autodownload (bool): Attempt to download dataset if not found locally. Default is False. Note: Download *.zip files from https://github.com/ultralytics/hub/tree/main/example_datasets i.e. https://github.com/ultralytics/hub/raw/main/example_datasets/coco8.zip for coco8.zip. Examples: >>> from ultralytics.data.utils import HUBDatasetStats >>> stats = HUBDatasetStats("path/to/coco8.zip", task="detect") # detect dataset >>> stats = HUBDatasetStats("path/to/coco8-seg.zip", task="segment") # segment dataset >>> stats = HUBDatasetStats("path/to/coco8-pose.zip", task="pose") # pose dataset >>> stats = HUBDatasetStats("path/to/dota8.zip", task="obb") # OBB dataset >>> stats = HUBDatasetStats("path/to/imagenet10.zip", task="classify") # classification dataset >>> stats.get_json(save=True) >>> stats.process_images() coco8.yamldetectFc Cst|}td|d||_|jdkr$t|}t|}||d<n4|t|\}}}zt|}d|d<t ||t ||}||d<Wnt yW} zt d| d} ~ wwt|dd|_ |j d |_ t|d t|d d |_||_dS) zInitialize class.z Starting HUB dataset checks for z....classifyrrJzerror/HUB/dataset_stats/initNz-hubr4r;)rr;)rr0r r`taskrr_unziprrrgr\hub_dirim_dirrPr-valuesstatsrC) selfrrr^ unzip_dirrCrmr yaml_pathrr@r@rA__init__Ms,          zHUBDatasetStats.__init__cCsVt|ds dd|fSt||jd}|s"Jd|d|ddt|t|fS) zUnzip data.zip.rHFNr,zError unzipping z, z6 not found. path/to/abc.zip MUST unzip to path/to/abc/T)rzrYrrQror1)rrr@r@rArhs  zHUBDatasetStats._unzipcCst||jt|jdS)z*Saves a compressed image for HUB previews.N)compress_one_imagerrrb)rrir@r@rA_hub_opssszHUBDatasetStats._hub_opsc s@fdddD]}dj|<j|}|durqddt|dD}|s)qjdkrud d lm}|j|}t t |j  t }|jD] } || d d 7<qIt ||d t |d |d dd|jDdj|<qd dlm} | j|jjd}tfddt|jt |ddD}t ||d d t |t t|d kd |d kd d fddt|j|jDdj|<q|rjjdddjd} td| dt| ddd} t j| Wdn 1s wY|rttj!jddd jS)!z(Return dataset JSON for Ultralytics HUB.csjdkr |d}n8jdvrdd|dD}n)jdkr9|dj\}}}t|d|d|||fd }n td jd t|d |}d d|DS)z:Update labels to integer class and 4 decimal place floats.rbboxes>obbsegmentcSsg|]}|qSr@)flattenr:r@r@rArBrz._round..rposerr6zUndefined dataset task=r7rcSs,g|]\}}t|dgdd|DqS)rcss|] }tt|dVqdS)N)r\rr:r@r@rArszFHUBDatasetStats.get_json.._round...)r)r;rrr@r@rArBr)rrr]rr ValueErrorzip)r5 coordinatesnnkrzippedrr@rA_roundzs    &z(HUBDatasetStats.get_json.._roundrqNcSrurvrwr+r@r@rArBryz,HUBDatasetStats.get_json..rzrr) ImageFolderr6)total per_class)r unlabelledrcSsg|] \}}t|j|iqSr@rrbr;rarr@r@rArBs)instance_stats image_statsr5 YOLODataset)img_pathrCrcs.g|]}tj|dtjddqS)rr) minlength)r]bincountrrrrC)r;rrr@rArBs  Statisticsrdesccs"g|] \}}t|j|iqSr@rr)rr@rArBrTparentsexist_okz stats.jsonzSaving rtrrrrF)indent sort_keys)"rrCrrr/rtorchvision.datasetsrr]rrPrrrimgstolistultralytics.datarrr r5rrWrim_filesrmkdirr r`r0rZjsondumpdumps) rrverboserrrarr]r<rr stats_pathrir@)rrrAget_jsonwsV          zHUBDatasetStats.get_jsonc Csddlm}|jjddddD]>}|j|durq||j||jd}tt}t| |j |j t ||dd D]}q>> from pathlib import Path >>> from ultralytics.data.utils import compress_one_image >>> for f in Path("path/to/dataset").rglob("*.jpg"): >>> compress_one_image(f) rrT)roptimizezHUB ops PIL failure rNr) interpolation)rrZrrrrrrr\r rQrimreadr INTER_AREAimwriterz) rif_newmax_dimrrrer im_heightim_widthr@r@rArs  " &"rcCs2ddl}|tjt|dd}||S)z1Load an Ultralytics *.cache dictionary from path.rNT) allow_pickle)gcdisabler]loadrzitemenable)rrcacher@r@rAload_dataset_cache_files rcCs||d<t|jr:|r|tt|d}t||Wdn1s)wYt |d|dSt |d|jddS)z9Save an Ultralytics dataset *.cache dictionary x to path.versionwbNzNew cache created: zCache directory z# is not writeable, cache not saved.) rrQrFunlinkrZrzr]rr r`rQ)rerr<rrr@r@rAsave_dataset_cache_files r)rGrHrIrJ)r6r6)r6)T)rJ)NrrH)ErxrrCrRr|rUrOmultiprocessing.poolrpathlibrtarfilerrnumpyr]PILrrultralytics.nn.autobackendrultralytics.utilsrr r r r r rrrrrrultralytics.utils.checksrrrultralytics.utils.downloadsrrrultralytics.utils.opsrHELP_URLr VID_FORMATSrzgetenvrr2rrFrvrrrrr rrr'r1rgrrrrrr@r@r@rAsR    8  L I ,    _\ "