o W h߇@sddlZddlZddlZddlZddlZddlZddlZddlm m Z ddl m Z ddlmZGdddejZdQddZdRd d Zd dZdSddZ         dTddZddZddZdUddZd d!Zd"d#ZdVd$d%ZdWd'd(Zd)d*Zd+d,Zd-d.Z d/d0Z!d1d2Z"d3d4Z#d5d6Z$dXd8d9Z%d:d;Z&dYdd?Z(dZd@dAZ)d[dBdCZ*dDdEZ+d\dGdHZ,dIej-dJej.fdKdLZ/dMdNZ0dOdPZ1dS)]N)LOGGER) batch_probiouc@sBeZdZdZddejfddZddZd d Zd d Z d dZ dS)Profilea YOLOv8 Profile class. Use as a decorator with @Profile() or as a context manager with 'with Profile():'. Attributes: t (float): Accumulated time. device (torch.device): Device used for model inference. cuda (bool): Whether CUDA is being used. Examples: >>> from ultralytics.utils.ops import Profile >>> with Profile(device=device) as dt: ... pass # slow operation here >>> print(dt) # prints "Elapsed time is 9.5367431640625e-07 s" NdevicecCs(||_||_t|ot|d|_dS)z Initialize the Profile class. Args: t (float): Initial time. device (torch.device): Device used for model inference. cudaN)trboolstr startswithr)selfrrr I/var/www/vscode/kcb/lib/python3.10/site-packages/ultralytics/utils/ops.py__init__!szProfile.__init__cCs||_|S)z Start timing.)timestartr r r r __enter__-s zProfile.__enter__cCs$||j|_|j|j7_dS)z Stop timing.N)rrdtr)r typevalue tracebackr r r__exit__2szProfile.__exit__cCsd|jdS)zZReturns a human-readable string representing the accumulated elapsed time in the profiler.zElapsed time is z s)rrr r r__str__7szProfile.__str__cCs|jr tj|jtS)zGet current time.)rtorch synchronizerr perf_counterrr r rr;sz Profile.time)rN) __name__ __module__ __qualname____doc__rrrrrrrr r r rrs  rcCs|j\}}t|dk|dk||k||kgdkr.|d|}|d|}|dk|dk@||k@||k@}||}||}t|r^tj||||g|jdStj d|jdS)ah Convert 1 segment label to 1 box label, applying inside-image constraint, i.e. (xy1, xy2, ...) to (xyxy). Args: segment (torch.Tensor): The segment label. width (int): The width of the image. height (int): The height of the image. Returns: (np.ndarray): The minimum and maximum x and y values of the segment. rdtype) Tnparrayminmaxsumclipanyr$zeros)segmentwidthheightxyinsider r r segment2boxBs :   (r5TFcCs|dur6t|d|d|d|d}t|d|d|ddt|d|d|ddf}n |dd}|d}|rl|d|d8<|d|d8<|sl|d|d8<|d |d8<|d dd f|<t||S) av Rescale bounding boxes from img1_shape to img0_shape. Args: img1_shape (tuple): The shape of the image that the bounding boxes are for, in the format of (height, width). boxes (torch.Tensor): The bounding boxes of the objects in the image, in the format of (x1, y1, x2, y2). img0_shape (tuple): The shape of the target image, in the format of (height, width). ratio_pad (tuple): A tuple of (ratio, pad) for scaling the boxes. If not provided, the ratio and pad will be calculated based on the size difference between the two images. padding (bool): If True, assuming the boxes is based on image augmented by yolo style. If False then do regular rescaling. xywh (bool): The box format is xywh or not. Returns: (torch.Tensor): The scaled bounding boxes, in the format of (x1, y1, x2, y2). Nrg?.r.r6.r7.r".r%)r)round clip_boxes) img1_shapeboxes img0_shape ratio_padpaddingxywhgainpadr r r scale_boxes]s"  rFcCs*t|tjr t|}t|||S)a Returns the nearest number that is divisible by the given divisor. Args: x (int): The number to make divisible. divisor (int | torch.Tensor): The divisor. Returns: (int): The nearest number divisible by the divisor. ) isinstancerTensorintr*mathceil)r2divisorr r rmake_divisibles rM?c Cstj|dd}||}t||}|r+|jdd}t||kddkd}||S|jd}tj||j d dd d|}tj||j d dd |d} || k} || }d|||kddk<t ||jdj }||S)a  NMS for oriented bounding boxes using probiou and fast-nms. Args: boxes (torch.Tensor): Rotated bounding boxes, shape (N, 5), format xywhr. scores (torch.Tensor): Confidence scores, shape (N,). threshold (float): IoU threshold. use_triu (bool): Whether to use `torch.triu` operator. It'd be useful for disable it when exporting obb models to some formats that do not support `torch.triu`. Returns: (torch.Tensor): Indices of boxes to keep after NMS. T descendingr6)diagonalrr)rargsortrtriu_nonzeror+squeeze_shapearangerviewexpandtopkindices) r?scores thresholduse_triu sorted_idxiouspicknrow_idxcol_idx upper_maskr r r nms_rotateds    rh?r ,皙?0uc*s4ddl}dkrdksnJddd|kr!dks*nJd|dtttfr5ddurAtjjdjdd ksJ|rcfd d D}durafd d |D}|Sjd}|ppjdd }jd|d }d |}ddd |fdk}t fdd |Dd}d| |}||dkM} dd| s| rt ddd fddd f<ntj t ddd fdd dffddt }tjdd |fjdg|}tjdjdg|}tt|D]\}\}}||}||||}}|rct||rc| sc||}tjt|||d f|jd}t |ddddf|dddd f<d|tt||dddfd f<t ||fd}|jdsjq|d ||fd\} }!}"|rt|!k\}#}$t | |#||#d |$df|$dddf|"|#fd}||#}n"|!jddd\}%}$|%dk}t | |%|$|"fd|}||}dur|dddd fkd}||||}}|jd}&|&sq|&| kr |ddd fjddd| }||||}}|dddd f|rdn| }'|ddd f}(| rQtj |ddddf|'|dddd f|ddddffdd})t|)|(|}#n|dddd f|'})|j|)|(|}#|#d}#||#||#d||<||<t ||krt d|ddnq|r||fS|S)a Perform non-maximum suppression (NMS) on a set of boxes, with support for masks and multiple labels per box. Args: prediction (torch.Tensor): A tensor of shape (batch_size, num_classes + 4 + num_masks, num_boxes) containing the predicted boxes, classes, and masks. The tensor should be in the format output by a model, such as YOLO. conf_thres (float): The confidence threshold below which boxes will be filtered out. Valid values are between 0.0 and 1.0. iou_thres (float): The IoU threshold below which boxes will be filtered out during NMS. Valid values are between 0.0 and 1.0. classes (List[int]): A list of class indices to consider. If None, all classes will be considered. agnostic (bool): If True, the model is agnostic to the number of classes, and all classes will be considered as one. multi_label (bool): If True, each box may have multiple labels. labels (List[List[Union[int, float, torch.Tensor]]]): A list of lists, where each inner list contains the apriori labels for a given image. The list should be in the format output by a dataloader, with each label being a tuple of (class_index, x1, y1, x2, y2). max_det (int): The maximum number of boxes to keep after NMS. nc (int): The number of classes output by the model. Any indices after this will be considered masks. max_time_img (float): The maximum time (seconds) for processing one image. max_nms (int): The maximum number of boxes into torchvision.ops.nms(). max_wh (int): The maximum box width and height in pixels. in_place (bool): If True, the input prediction tensor will be modified in place. rotated (bool): If Oriented Bounding Boxes (OBB) are being passed for NMS. end2end (bool): If the model doesn't require NMS. return_idxs (bool): Return the indices of the detections that were kept. Returns: (List[torch.Tensor]): A list of length batch_size, where each element is a tensor of shape (num_boxes, 6 + num_masks) containing the kept boxes, with columns (x1, y1, x2, y2, confidence, class, mask1, mask2, ...). rNr6zInvalid Confidence threshold z&, valid values are between 0.0 and 1.0z Invalid IoU rSrRcs,g|]}||dddfkdqS)Nr%r .0pred) conf_thresmax_detr r s,z'non_max_suppression..cs.g|]}||ddddfkdqS)Nrnr6)r-ro)classesr rrts.r%cs g|] }tjt|jdqS)rS)rrYlenrrpi) predictionr rrts ).Ng@.dim)rr6rug?T)keepdimrOr7zNMS time limit z.3fz s exceeded)! torchvisionrGlisttuplertensorrrXamaxstack transpose xywh2xyxycatrr. enumerateziprwrangelongsplitwherefloatr*rZr-rTrhopsnmsreshaperwarning)*rzrr iou_thresrvagnostic multi_labellabelsrsnc max_time_imgmax_nmsmax_whin_placerotatedend2end return_idxsroutputbsnmmixcxinds time_limitrkeepixir2xkfiltlbvboxclsmaskryjconfrdcr^r?r )rvrrrsrzrnon_max_suppressions3&&    ". (( :     "H  rcCst|tjr8|dd|d|d<|dd|d|d<|dd|d|d<|dd|d|d<|S|dddgfd|d|dddgf<|ddd gfd|d|ddd gf<|S) a< Takes a list of bounding boxes and a shape (height, width) and clips the bounding boxes to the shape. Args: boxes (torch.Tensor | numpy.ndarray): The bounding boxes to clip. shape (tuple): The shape of the image. Returns: (torch.Tensor | numpy.ndarray): The clipped boxes. r8rr6r9r:r;.r7r"rGrrHclampr,)r?rXr r rr=Ys ((r=cCstt|tjr |dd|d|d<|dd|d|d<|S|dd|d|d<|dd|d|d<|S)aF Clip line coordinates to the image boundaries. Args: coords (torch.Tensor | numpy.ndarray): A list of line coordinates. shape (tuple): A tuple of integers representing the size of the image in the format (height, width). Returns: (torch.Tensor | numpy.ndarray): Clipped coordinates. r8rr6r9r)coordsrXr r r clip_coordsos rc Cs@|j}|dd|ddkr|S|dur?t|d|d|d|d}|d|d|d|d|d|df}n|d}t|dt|d}}t|d|dt|d|d}} t|jdkrvtdt|j||||| f}t||d|df}t|jdkr|dddddf}|S)a Takes a mask, and resizes it to the original image size. Args: masks (np.ndarray): Resized and padded masks/images, [h, w, num]/[h, w, 3]. im0_shape (tuple): The original image shape. ratio_pad (tuple): The ratio of the padding to the original image. Returns: masks (np.ndarray): The masks that are being returned with shape [h, w, num]. Nr7rr6z/"len of masks shape" should be 2 or 3, but got )rXr)rIrw ValueErrorcv2resize) masks im0_shaperA im1_shaperDrEtopleftbottomrightr r r scale_images  "2*rcCs|jddksJd|jt|}|d|dd|d<|d|dd|d<|d|d|d<|d|d|d<|S) a Convert bounding box coordinates from (x1, y1, x2, y2) format to (x, y, width, height) format where (x1, y1) is the top-left corner and (x2, y2) is the bottom-right corner. Args: x (np.ndarray | torch.Tensor): The input bounding box coordinates in (x1, y1, x2, y2) format. Returns: y (np.ndarray | torch.Tensor): The bounding box coordinates in (x, y, width, height) format. rRr%9input shape last dimension expected 4 but input shape is r8r:r7r9r;rX empty_liker2r3r r r xyxy2xywhs rcCsv|jddksJd|jt|}|dddf}|dddfd}|||dddf<|||dddf<|S)a Convert bounding box coordinates from (x, y, width, height) format to (x1, y1, x2, y2) format where (x1, y1) is the top-left corner and (x2, y2) is the bottom-right corner. Note: ops per 2 channels faster than per channel. Args: x (np.ndarray | torch.Tensor): The input bounding box coordinates in (x, y, width, height) format. Returns: y (np.ndarray | torch.Tensor): The bounding box coordinates in (x1, y1, x2, y2) format. rRr%r.Nr7r)r2r3xywhr r rrs rcCs|jddksJd|jt|}||d|dd||d<||d|dd||d<||d|dd||d<||d|dd||d<|S) a  Convert normalized bounding box coordinates to pixel coordinates. Args: x (np.ndarray | torch.Tensor): The bounding box coordinates. w (int): Width of the image. h (int): Height of the image. padw (int): Padding width. padh (int): Padding height. Returns: y (np.ndarray | torch.Tensor): The coordinates of the bounding box in the format [x1, y1, x2, y2] where x1,y1 is the top-left corner, x2,y2 is the bottom-right corner of the bounding box. rRr%rr8r:r7r9r;r)r2whpadwpadhr3r r r xywhn2xyxys    rrcCs|r t|||||f}|jddksJd|jt|}|d|dd||d<|d|dd||d<|d|d||d<|d|d||d<|S) a Convert bounding box coordinates from (x1, y1, x2, y2) format to (x, y, width, height, normalized) format. x, y, width and height are normalized to image dimensions. Args: x (np.ndarray | torch.Tensor): The input bounding box coordinates in (x1, y1, x2, y2) format. w (int): The width of the image. h (int): The height of the image. clip (bool): If True, the boxes will be clipped to the image boundaries. eps (float): The minimum value of the box's width and height. Returns: y (np.ndarray | torch.Tensor): The bounding box coordinates in (x, y, width, height, normalized) format rRr%rr8r:r7r9r;)r=rXr)r2rrr,epsr3r r r xyxy2xywhnsrcCsRt|tjr |nt|}|d|dd|d<|d|dd|d<|S)aX Convert the bounding box format from [x, y, w, h] to [x1, y1, w, h], where x1, y1 are the top-left coordinates. Args: x (np.ndarray | torch.Tensor): The input tensor with the bounding box coordinates in the xywh format Returns: y (np.ndarray | torch.Tensor): The bounding box coordinates in the xyltwh format r8r:r7r9r;rGrrHcloner'copyrr r r xywh2ltwh rcCsJt|tjr |nt|}|d|d|d<|d|d|d<|S)aU Convert nx4 bounding boxes from [x1, y1, x2, y2] to [x1, y1, w, h], where xy1=top-left, xy2=bottom-right. Args: x (np.ndarray | torch.Tensor): The input tensor with the bounding boxes coordinates in the xyxy format Returns: y (np.ndarray | torch.Tensor): The bounding box coordinates in the xyltwh format. r:r8r;r9rrr r r xyxy2ltwh rcCsRt|tjr |nt|}|d|dd|d<|d|dd|d<|S)z Convert nx4 boxes from [x1, y1, w, h] to [x, y, w, h] where xy1=top-left, xy=center. Args: x (torch.Tensor): the input tensor Returns: y (np.ndarray | torch.Tensor): The bounding box coordinates in the xywh format. r8r:r7r9r;rrr r r ltwh2xywh rrc Cst|tj}|r|n|}|t|dd}g}|D]}t|\\}}\}}} | ||||| dt j gq|rGtj ||j |jdSt |S)a Convert batched Oriented Bounding Boxes (OBB) from [xy1, xy2, xy3, xy4] to [xywh, rotation]. Rotation values are returned in radians from 0 to pi/2. Args: x (numpy.ndarray | torch.Tensor): Input box corners [xy1, xy2, xy3, xy4] of shape (n, 8). Returns: (numpy.ndarray | torch.Tensor): Converted data in [cx, cy, w, h, rotation] format of shape (n, 5). rRr7rr$)rGrrHcpunumpyrrwr minAreaRectappendr'pirrr$asarray) r2is_torchpointsrboxesptscxcyrrangler r rxyxyxyxy2xywhr0s  "rcsttjrtjtjtjtjfn tjtjtjtjf\}}}}dddf}fddt ddD\}}}||||} } |d| |d| g} | d| |d| g} || d} || d} || | } || | }|| | }|| | }|| |||gdS) a Convert batched Oriented Bounding Boxes (OBB) from [xywh, rotation] to [xy1, xy2, xy3, xy4]. Rotation values should be in radians from 0 to pi/2. Args: x (numpy.ndarray | torch.Tensor): Boxes in [cx, cy, w, h, rotation] format of shape (n, 5) or (b, n, 5). Returns: (numpy.ndarray | torch.Tensor): Converted corner points of shape (n, 4, 2) or (b, n, 4, 2). .Nr7c3s$|] }d||dfVqdS).r6Nr rxr2r r Ys"z!xywhr2xyxyxyxy..rurRr{) rGrrHcossinrrr' concatenater)r2rrrrctrrrr cos_value sin_valuevec1vec2pt1pt2pt3pt4r rrxywhr2xyxyxyxyGs        rcCsJt|tjr |nt|}|d|d|d<|d|d|d<|S)a Convert bounding box from [x1, y1, w, h] to [x1, y1, x2, y2] where xy1=top-left, xy2=bottom-right. Args: x (np.ndarray | torch.Tensor): The input image. Returns: (np.ndarray | torch.Tensor): The xyxy coordinates of the bounding boxes. r:r8r;r9rrr r r ltwh2xyxyfrrcCsHg}|D]}|j\}}|||||gqtt|S)a+ Convert segment labels to box labels, i.e. (cls, xy1, xy2, ...) to (cls, xywh). Args: segments (list): List of segments, each segment is a list of points, each point is a list of x, y coordinates. Returns: (np.ndarray): The xywh coordinates of the bounding boxes. )r&rr)r*rr'r()segmentsr?sr2r3r r rsegments2boxesvs   $rcst|D]g\}t|krqtjddddffddtdtdt|kr4|tn|ttt|krOttntjfddtdDtj d dd j ||<q|S) aT Inputs a list of segments (n,2) and returns a list of segments (n,2) up-sampled to n points each. Args: segments (list): A list of (n,2) arrays, where n is the number of points in the segment. n (int): Number of points to resample the segment to. Returns: segments (list): The resampled segments. rr6N)axisc s&g|]}tdd|fqSN)r'interprxrr2xpr rrts&z%resample_segments..r7r#rR) rrwr'rlinspacerYinsert searchsortedrfloat32rr&)rrdryr rrresample_segmentss ".&.rc Cs|j\}}}t|dddddfdd\}}}}tj||j|jdddddf} tj||j|jdddddf} || |k| |k| |k| |kS)z Crop masks to bounding boxes. Args: masks (torch.Tensor): [n, h, w] tensor of masks. boxes (torch.Tensor): [n, 4] tensor of bbox coordinates in relative point form. Returns: (torch.Tensor): Cropped masks. Nr%r6r)rXrchunkrYrr$) rr?_rrx1y1x2y2rrr r r crop_masks (""$r cCs|j\}}}|\}} |||dd||} || } ||} |} | dddf| 9<| dddf| 9<| dddf| 9<| dddf| 9<t| | } |ritj| d|ddd d} | d S) a) Apply masks to bounding boxes using the output of the mask head. Args: protos (torch.Tensor): A tensor of shape [mask_dim, mask_h, mask_w]. masks_in (torch.Tensor): A tensor of shape [n, mask_dim], where n is the number of masks after NMS. bboxes (torch.Tensor): A tensor of shape [n, 4], where n is the number of masks after NMS. shape (tuple): A tuple of integers representing the size of the input image in the format (h, w). upsample (bool): A flag to indicate whether to upsample the mask to the original image size. Returns: (torch.Tensor): A binary mask tensor of shape [n, h, w], where n is the number of masks after NMS, and h and w are the height and width of the input image. The mask is applied to the bounding boxes. rRNrr7r"r6bilinearFmode align_cornersr)rXrrZrr F interpolategt_)protosmasks_inbboxesrXupsamplermhmwihiwr width_ratio height_ratiodownsampled_bboxesr r r process_masks   r cCsP|j\}}}|||dd||}t|d|d}t||}|dS)a Apply masks to bounding boxes using the output of the mask head with native upsampling. Args: protos (torch.Tensor): [mask_dim, mask_h, mask_w]. masks_in (torch.Tensor): [n, mask_dim], n is number of masks after nms. bboxes (torch.Tensor): [n, 4], n is number of masks after nms. shape (tuple): The size of the input image (h,w). Returns: (torch.Tensor): The returned masks with dimensions [h, w, n]. rRNrr)rXrrZ scale_masksr r)rrrrXrrrrr r rprocess_mask_natives   r"c Cs|jdd\}}t||d||d}||d|||d|g}|r8|dd<|dd<|rFt|dt|dfnd\}}t||dt||d} } |d|| || f}tj||ddd }|S) aN Rescale segment masks to shape. Args: masks (torch.Tensor): (N, C, H, W). shape (tuple): Height and width. padding (bool): If True, assuming the boxes is based on image augmented by yolo style. If False then do regular rescaling. Returns: (torch.Tensor): Rescaled masks. r7Nrr6)rr.rFr)rXr)rIrr) rrXrBrrrDrErrrrr r rr!s  $"r!cCs|dur.t|d|d|d|d}|d|d|d|d|d|df}n |dd}|d}|rN|d|d8<|d|d8<|d|<|d|<t||}|ry|d|d<|d|d<|S)a Rescale segment coordinates (xy) from img1_shape to img0_shape. Args: img1_shape (tuple): The shape of the image that the coords are from. coords (torch.Tensor): The coords to be scaled of shape n,2. img0_shape (tuple): The shape of the image that the segmentation is being applied to. ratio_pad (tuple): The ratio of the image size to the padded image size. normalize (bool): If True, the coordinates will be normalized to the range [0, 1]. padding (bool): If True, assuming the boxes is based on image augmented by yolo style. If False then do regular rescaling. Returns: coords (torch.Tensor): The scaled coordinates. Nrr6r7r8r9)r)r)r>rr@rA normalizerBrDrEr r r scale_coordss"2  r$c Csl|jdd\}}}}}|tjtjdk}t|||}t|||}|tjd}tj|||||gddS)z Regularize rotated boxes in range [0, pi/2]. Args: rboxes (torch.Tensor): Input boxes of shape(N, 5) in xywhr format. Returns: (torch.Tensor): The regularized boxes. rRr|r7)unbindrJrrrr) rr2r3rrrswapw_h_r r rregularize_rboxes%s  r)allcCsddlm}g}|dD]U}t|tjtj d}|r[|dkrAt |dkr8t |dd|Dn|d dd }n|d krZt |t d d|D dd }nt d }||d q|S)z Convert masks to segments. Args: masks (torch.Tensor): The output of the model, which is a tensor of shape (batch_size, 160, 160). strategy (str): 'all' or 'largest'. Returns: (list): List of segment masks. r)merge_multi_segmentuint8r*r6cSsg|]}|ddqS)rRr7)rrpr2r r rrtKsz"masks2segments..rRr7largestcSsg|]}t|qSr )rwr-r r rrtPs)rr7r)ultralytics.data.converterr+rIrrastyper findContours RETR_EXTERNALCHAIN_APPROX_SIMPLErwr'rrr(argmaxr.r)rstrategyr+rr2rr r rmasks2segments8s  * r6batchreturncCs0|dddddddtjS)as Convert a batch of FP32 torch tensors (0.0-1.0) to a NumPy uint8 array (0-255), changing from BCHW to BHWC layout. Args: batch (torch.Tensor): Input tensor batch of shape (Batch, Channels, Height, Width) and dtype torch.float32. Returns: (np.ndarray): Output NumPy array batch of shape (Batch, Height, Width, Channels) and dtype uint8. rr7r"r6)permute contiguousrtorr,rr)r7r r rconvert_torch2numpy_batchWs0 r=cCstjdd|dS)z Cleans a string by replacing special characters with '_' character. Args: s (str): A string needing special characters replaced. Returns: (str): A string with special characters replaced by an underscore _. u"[|@#!¡·$€%&()=?¿^*;:,¨´><+]r)patternreplstring)resub)rr r r clean_strds rCcCs,t|tjrtj|tjdStj|tjdS)zTCreates empty torch.Tensor or np.ndarray with same shape as input and float32 dtype.r#)rGrrHrrr'rr r rrqsr)r!r!)NTF)rNT)rirNNFFr rjrrkrlrmTFFFr)r!r!rr)r!r!Fr)r)F)T)NFT)r*)2 contextlibrJrArrrr'rtorch.nn.functionalnn functionalrultralytics.utilsrultralytics.utils.metricsrContextDecoratorrr5rFrMrhrr=rrrrrrrrrrrrrrr r r"r!r$r)r6rHndarrayr=rCrr r r rsr   1 % % % #    !  #