o V h#@shddlZddlmZddlmZddlmZmZddlm Z ddl m Z ddl m Z Gd d d eZdS) N)Image)SegmentationPredictor) DEFAULT_CFGchecks)box_iou) scale_masks)adjust_bboxes_to_image_bordercsNeZdZdZeddffdd ZfddZd ddZd d Zd d Z Z S)FastSAMPredictora[ FastSAMPredictor is specialized for fast SAM (Segment Anything Model) segmentation prediction tasks. This class extends the SegmentationPredictor, customizing the prediction pipeline specifically for fast SAM. It adjusts post-processing steps to incorporate mask prediction and non-maximum suppression while optimizing for single-class segmentation. Attributes: prompts (dict): Dictionary containing prompt information for segmentation (bboxes, points, labels, texts). device (torch.device): Device on which model and tensors are processed. clip_model (Any, optional): CLIP model for text-based prompting, loaded on demand. clip_preprocess (Any, optional): CLIP preprocessing function for images, loaded on demand. Methods: postprocess: Applies box postprocessing for FastSAM predictions. prompt: Performs image segmentation inference based on various prompt types. _clip_inference: Performs CLIP inference to calculate similarity between images and text prompts. set_prompts: Sets prompts to be used during inference. Ncst|||i|_dS)ac Initialize the FastSAMPredictor with configuration and callbacks. This initializes a predictor specialized for Fast SAM (Segment Anything Model) segmentation tasks. The predictor extends SegmentationPredictor with custom post-processing for mask prediction and non-maximum suppression optimized for single-class segmentation. Args: cfg (dict): Configuration for the predictor. Defaults to Ultralytics DEFAULT_CFG. overrides (dict, optional): Configuration overrides. _callbacks (list, optional): List of callback functions. N)super__init__prompts)selfcfg overrides _callbacks __class__V/var/www/vscode/kcb/lib/python3.10/site-packages/ultralytics/models/fastsam/predict.pyr #s zFastSAMPredictor.__init__c s|jdd}|jdd}|jdd}|jdd}t|||}|D]:} tjdd| jd| jdg|djtjd} t | j j | j} t t | d| d k} | dkr`| | j j | <q&|j|||||d S) a Apply postprocessing to FastSAM predictions and handle prompts. Args: preds (List[torch.Tensor]): Raw predictions from the model. img (torch.Tensor): Input image tensor that was fed to the model. orig_imgs (List[numpy.ndarray]): Original images before preprocessing. Returns: (List[Results]): Processed results with prompts applied. bboxesNpointslabelstextsrr)devicedtypeg?)rrrr)r popr postprocesstorchtensor orig_shaperfloat32r boxesxyxynonzerorflattennumelprompt) rpredsimg orig_imgsrrrrresultsresultfull_boxr"idxrrrr3s "  zFastSAMPredictor.postprocessc s|dur|dur|dur|Sg}t|ts|g}|D]}t|dkr)||q|jjjdd|jkrAtd|jdt j t|t j |j d}|durt j |t j|j d}|jdkrd|dn|}|dddf|dddf|dddf|dddf} t fdd|D} t jd d } | dddf| | } d |t j| | dd <|durg|]}dd|d|d|d|dfjddqS)Nrr/rr0rr0dim)sum).0bmasksrr os>z+FastSAMPredictor.prompt..r1r2Tz4Excepted `labels` got same size as `point`, but got z and )as_tuplecss|]}t|VqdS)N)int)r5xrrr sz*FastSAMPredictor.prompt..dr)! isinstancelistlenappendr8datashaper rrzerosboolr as_tensorint32ndimstackr4argmaxoneszipr$str enumerater"r#tolistr fromarrayorig_img_clip_inferencerr;)rr+rrrrprompt_resultsr,r. bbox_areas mask_areasfull_mask_areasunion point_idxpointlabelcrop_ims filter_idxir6x1y1x2y2 similaritytext_idxrr7rr'Osl    @ 4    .  &zFastSAMPredictor.promptcszddl}Wntytdddl}Ynwtdr#tds/|jdjd\__t fdd |D}| | j}j |}j|}||jd d d }||jd d d }||dddfd S) a Perform CLIP inference to calculate similarity between images and text prompts. Args: images (List[PIL.Image]): List of source images, each should be PIL.Image with RGB channel order. texts (List[str]): List of prompt texts, each should be a string object. Returns: (torch.Tensor): Similarity matrix between given images and texts with shape (M, N). rNz+git+https://github.com/ultralytics/CLIP.git clip_modelclip_preprocesszViT-B/32r@csg|] }|jqSr)rhtor)r5imagerrrr9sz4FastSAMPredictor._clip_inference..r?T)r3keepdim)clip ImportErrorrcheck_requirementshasattrloadrrgrhrrLtokenizeri encode_image encode_textnormr4)rimagesrrmtokenized_textimage_features text_featuresrrkrrUs      z FastSAMPredictor._clip_inferencecCs ||_dS)z(Set prompts to be used during inference.N)r )rr rrr set_promptss zFastSAMPredictor.set_prompts)NNNN) __name__ __module__ __qualname____doc__rr rr'rUrz __classcell__rrrrr s  Jr )rPILrultralytics.models.yolo.segmentrultralytics.utilsrrultralytics.utils.metricsrultralytics.utils.opsrutilsr r rrrrs