o W h@sddlmZddlZddlmmZddlmZddlmZddl m Z ddl m Z ddl mZdd lmZmZdd lmZmZdd lmZmZd ZGd ddejZGdddejjZdS))ListN)nn) trunc_normal_)MLP)LOGGER)SAM2TwoWayTransformer) MaskDecoderSAM2MaskDecoder)ImageEncoderViT PromptEncoder)get_1d_sine_peselect_closest_cond_framesgc s^eZdZUdZdZeed<  ddedede d e ed e ed d f fd d Z ddZ Z S)SAMModela Segment Anything Model (SAM) for object segmentation tasks. This class combines image encoders, prompt encoders, and mask decoders to predict object masks from images and input prompts. Attributes: mask_threshold (float): Threshold value for mask prediction. image_encoder (ImageEncoderViT): Backbone for encoding images into embeddings. prompt_encoder (PromptEncoder): Encoder for various types of input prompts. mask_decoder (MaskDecoder): Predicts object masks from image and prompt embeddings. Methods: __init__: Initializes the SAMModel with encoders, decoder, and normalization parameters. Examples: >>> image_encoder = ImageEncoderViT(...) >>> prompt_encoder = PromptEncoder(...) >>> mask_decoder = MaskDecoder(...) >>> sam_model = SAMModel(image_encoder, prompt_encoder, mask_decoder) >>> # Further usage depends on SAMPredictor class Notes: All forward() operations are implemented in the SAMPredictor class. mask_thresholdg33333^@gR]@gRY@g(\2M@g(\L@gL@ image_encoderprompt_encoder mask_decoder pixel_mean pixel_stdreturnNcs\t||_||_||_|dt|dddd|dt|dddddS)a Initialize the SAMModel class to predict object masks from an image and input prompts. Args: image_encoder (ImageEncoderViT): The backbone used to encode the image into image embeddings. prompt_encoder (PromptEncoder): Encodes various types of input prompts. mask_decoder (MaskDecoder): Predicts masks from the image embeddings and encoded prompts. pixel_mean (List[float]): Mean values for normalizing pixels in the input image. pixel_std (List[float]): Std values for normalizing pixels in the input image. Examples: >>> image_encoder = ImageEncoderViT(...) >>> prompt_encoder = PromptEncoder(...) >>> mask_decoder = MaskDecoder(...) >>> sam_model = SAMModel(image_encoder, prompt_encoder, mask_decoder) >>> # Further usage depends on SAMPredictor class Notes: All forward() operations moved to SAMPredictor. rrFrN) super__init__rrrregister_buffertorchTensorview)selfrrrrr __class__V/var/www/vscode/kcb/lib/python3.10/site-packages/ultralytics/models/sam/modules/sam.pyr9s "zSAMModel.__init__cCsBt|jdr |j|||j_dd|D|j_|d|j_dS)z Set image size to make model compatible with different image sizes. Args: imgsz (Tuple[int, int]): The size of the input image. set_imgszcSg|]}|dqSr$.0xr$r$r% fz&SAMModel.set_imgsz..rN)hasattrrr&rinput_image_sizeimage_embedding_sizeimg_sizer!imgszr$r$r%r&\s  zSAMModel.set_imgsz)rr)__name__ __module__ __qualname____doc__rfloat__annotations__r r r rrr& __classcell__r$r$r"r%rs&  #rcs:eZdZUdZdZeed<                          d8d edededededededeffdd Ze ddZ ddZ ddZ d9ddZ dd Zd!ejfd"d#Zd$d%Z d:d&d'Zd(d)Zd*d+Zd,d-Z  d;d.d/Zd0d1Zed2d3Zd:d4d5Zd6d7ZZS)< SAM2Modela SAM2Model class for Segment Anything Model 2 with memory-based video object segmentation capabilities. This class extends the functionality of SAM to handle video sequences, incorporating memory mechanisms for temporal consistency and efficient tracking of objects across frames. Attributes: mask_threshold (float): Threshold value for mask prediction. image_encoder (ImageEncoderViT): Visual encoder for extracting image features. memory_attention (nn.Module): Module for attending to memory features. memory_encoder (nn.Module): Encoder for generating memory representations. num_maskmem (int): Number of accessible memory frames. image_size (int): Size of input images. backbone_stride (int): Stride of the backbone network output. sam_prompt_embed_dim (int): Dimension of SAM prompt embeddings. sam_image_embedding_size (int): Size of SAM image embeddings. sam_prompt_encoder (PromptEncoder): Encoder for processing input prompts. sam_mask_decoder (SAM2MaskDecoder): Decoder for generating object masks. obj_ptr_proj (nn.Module): Projection layer for object pointers. obj_ptr_tpos_proj (nn.Module): Projection for temporal positional encoding in object pointers. Methods: forward_image: Processes image batch through encoder to extract multi-level features. track_step: Performs a single tracking step, updating object masks and memory features. Examples: >>> model = SAM2Model(image_encoder, memory_attention, memory_encoder) >>> image_batch = torch.rand(1, 3, 512, 512) >>> features = model.forward_image(image_batch) >>> track_results = model.track_step(0, True, features, None, None, None, {}) rrr)?FrrTNuse_multimask_token_for_obj_ptrpred_obj_scorespred_obj_scores_mlpfixed_no_obj_ptrsoft_no_obj_ptruse_mlp_for_obj_ptr_projno_obj_embed_spatialcompile_image_encoderc$$srt||_| |_| rdnd|_||_||_|r%tjj ddddd|_ ||_ |r.|s.J||_ ||_ ||_||_|j|_||_|j|_t|jdr[t|jjdr[|jjjjd|_||_tjt|dd|j|_t|jdd tjtdd|j|_tjtdd|j|_t|jdd t|jdd | |_||_ ||_!| |_"||_#||_$| |_%||_&||_'||_(||_)||_*||_+||_,||_-|"|_.||_/||_0||_1||_2|j1r|j/sJ|jsJ|j/r|jrtjtd|j|_3t|j3dd | |_4d |_5|!rtjtd|j|_5t|j5dd |6| |_7|#r7t89d tj:|jj;d d dd|j_;d Sd S)a Initialize the SAM2Model for video object segmentation with memory-based tracking. Args: image_encoder (nn.Module): Visual encoder for extracting image features. memory_attention (nn.Module): Module for attending to memory features. memory_encoder (nn.Module): Encoder for generating memory representations. num_maskmem (int): Number of accessible memory frames. Default is 7 (1 input frame + 6 previous frames). image_size (int): Size of input images. backbone_stride (int): Stride of the image backbone output. sigmoid_scale_for_mem_enc (float): Scale factor for mask sigmoid probability. sigmoid_bias_for_mem_enc (float): Bias factor for mask sigmoid probability. binarize_mask_from_pts_for_mem_enc (bool): Whether to binarize sigmoid mask logits on interacted frames with clicks during evaluation. use_mask_input_as_output_without_sam (bool): Whether to directly output the input mask without using SAM prompt encoder and mask decoder on frames with mask input. max_cond_frames_in_attn (int): Maximum number of conditioning frames to participate in memory attention. -1 means no limit. directly_add_no_mem_embed (bool): Whether to directly add no-memory embedding to image feature on the first frame. use_high_res_features_in_sam (bool): Whether to use high-resolution feature maps in the SAM mask decoder. multimask_output_in_sam (bool): Whether to output multiple (3) masks for the first click on initial conditioning frames. multimask_min_pt_num (int): Minimum number of clicks to use multimask output in SAM. multimask_max_pt_num (int): Maximum number of clicks to use multimask output in SAM. multimask_output_for_tracking (bool): Whether to use multimask output for tracking. use_multimask_token_for_obj_ptr (bool): Whether to use multimask tokens for object pointers. iou_prediction_use_sigmoid (bool): Whether to use sigmoid to restrict IoU prediction to [0-1]. memory_temporal_stride_for_eval (int): Memory bank's temporal stride during evaluation. non_overlap_masks_for_mem_enc (bool): Whether to apply non-overlapping constraints on object masks in memory encoder during evaluation. use_obj_ptrs_in_encoder (bool): Whether to cross-attend to object pointers from other frames in the encoder. max_obj_ptrs_in_encoder (int): Maximum number of object pointers from other frames in encoder cross-attention. add_tpos_enc_to_obj_ptrs (bool): Whether to add temporal positional encoding to object pointers in the encoder. proj_tpos_enc_in_obj_ptrs (bool): Whether to add an extra linear projection layer for temporal positional encoding in object pointers. use_signed_tpos_enc_to_obj_ptrs (bool): Whether to use signed distance (instead of unsigned absolute distance) in the temporal positional encoding in the object pointers, only relevant when both `use_obj_ptrs_in_encoder=True` and `add_tpos_enc_to_obj_ptrs=True`. only_obj_ptrs_in_the_past_for_eval (bool): Whether to only attend to object pointers in the past during evaluation. pred_obj_scores (bool): Whether to predict if there is an object in the frame. pred_obj_scores_mlp (bool): Whether to use an MLP to predict object scores. fixed_no_obj_ptr (bool): Whether to have a fixed no-object pointer when there is no object present. soft_no_obj_ptr (bool): Whether to mix in no-object pointer softly for easier recovery and error mitigation. use_mlp_for_obj_ptr_proj (bool): Whether to use MLP for object pointer projection. no_obj_embed_spatial (bool): Whether add no obj embedding to spatial frames. sam_mask_decoder_extra_args (Dict | None): Extra arguments for constructing the SAM mask decoder. compile_image_encoder (bool): Whether to compile the image encoder for faster inference. Examples: >>> image_encoder = ImageEncoderViT(...) >>> memory_attention = SAM2TwoWayTransformer(...) >>> memory_encoder = nn.Sequential(...) >>> model = SAM2Model(image_encoder, memory_attention, memory_encoder) >>> image_batch = torch.rand(1, 3, 512, 512) >>> features = model.forward_image(image_batch) >>> track_results = model.track_step(0, True, features, None, None, None, {}) r) kernel_sizestrideout_projweightrg{Gz?)stdNzFImage encoder compilation is enabled. First forward pass will be slow.z max-autotuneTF)mode fullgraphdynamic)>> backbone_features = torch.rand(1, 256, 32, 32) >>> point_inputs = {"point_coords": torch.rand(1, 2, 2), "point_labels": torch.tensor([[1, 0]])} >>> mask_inputs = torch.rand(1, 1, 512, 512) >>> results = model._forward_sam_heads(backbone_features, point_inputs, mask_inputs) >>> ( ... low_res_multimasks, ... high_res_multimasks, ... ious, ... low_res_masks, ... high_res_masks, ... obj_ptr, ... object_score_logits, ... ) = results rrrrHN point_coords point_labelsr)dtyperrIFbilinearTsize align_cornersrO antialias)pointsboxesmasks)image_embeddingsimage_pesparse_prompt_embeddingsdense_prompt_embeddingsmultimask_output repeat_imagehigh_res_features)rrOrrdim)rrrrrrdonesint32lenrarmask_input_sizeF interpolater9r get_dense_perAwhere NO_OBJ_SCORErtargmaxarange unsqueezerrDsigmoidrCrw)r!backbone_features point_inputs mask_inputsrrBrsam_point_coordssam_point_labelssam_mask_promptsparse_embeddingsdense_embeddingslow_res_multimasksioussam_output_tokensobject_score_logitsis_obj_appearinghigh_res_multimaskssam_output_token best_iou_inds batch_inds low_res_maskshigh_res_masksobj_ptrlambda_is_obj_appearingr$r$r%_forward_sam_headss 6"(       zSAM2Model._forward_sam_headscCsd\}}|}|||}tj||dd|ddfdddd}||d d } |js@tj|d |j|j d } n|j || ||d \} } } } } } } tj | d d kd d} | d} | } || |}|jr|jrx| | } | d | |j} ||| ||| |fS)zFProcess mask inputs directly as output, bypassing SAM encoder/decoder.)g4@$rrIrFrTrrrr)rrrrr).N)r9rrrnew_onesrTrrdr^rrrWanyflattenrArCrw)r!rrr out_scaleout_biasmask_inputs_floatrrrr_rrrr$r$r%_use_mask_as_output$sD  zSAM2Model._use_mask_as_output img_batchcCsL||}|jr$|j|dd|dd<|j|dd|dd<|S)zRProcess image batch through encoder to extract multi-level features for SAM model. backbone_fpnrr)rrRrconv_s0conv_s1)r!r backbone_outr$r$r% forward_imageSs zSAM2Model.forward_imagecCst|dt|dksJt|d|jksJ|d|j d}|d|j d}dd|D}dd|D}dd|D}||||fS)zZPrepare and flatten visual features from the image backbone output for further processing.rvision_pos_encNcSs g|] }|jd|jdfqS)rr)rar*r$r$r%r-e z8SAM2Model._prepare_backbone_features..cS g|] }|ddddqSrrrrpermuter*r$r$r%r-grcSrrrr*r$r$r%r-hr)rrS)r!r feature_mapsvision_pos_embeds feat_sizes vision_featsr$r$r%_prepare_backbone_features]s z$SAM2Model._prepare_backbone_featuresc +s|dd} j} |d\} } |dj} jdkr)|dddd| | | | Sd}r/dnd|sgg}}t|ddksCJ|d}t|j\}}dd| D}j r^dnj }t djD]T}j|}|dkr}rx|n|}n%sd||}||d|}nd | |}||d|}|d |d }|d ur| |d }|||fqg|D]D\}}|d urq|d j| d d }||dddd|d dj| d}|dddd}|jj|d}||qjrt|j}j s"jr"fdd|D}n|}fdd|D}t d|D]9} r?| n| }!|!dksR|d urT|!|krTn|d |!| |!d }|d urn|| |dfq6|rt|\}"}#tj|#dd}$jr|d}%jr| nj}&tj|"| d}'t|'|%|&d}'|'}'|' d!d| j}'n |$"t|"| j}'j| kr|$#d| | jj}$|$dddddd}$|'j$| jdd}'||$||'|$j%d}n1d}n.j&r|dj'}(|(ddd| | | | }(|(Sj'!d| jg}j(!d| jg}tj)|dd})tj)|dd}*j*|||)|*|d}(|(ddd| | | | }(|(S)zePrepare memory-conditioned features by fusing current frame's visual features with previous memories.rrrrcond_frame_outputscSsg|]}d|fqS)rr$)r+outr$r$r%r-r.zBSAM2Model._prepare_memory_conditioned_features..non_cond_frame_outputsNmaskmem_featuresT)r non_blockingmaskmem_pos_encrcs,i|]\}}r |krn|kr||qSr$r$r+tr) frame_idxtrack_in_reverser$r% szBSAM2Model._prepare_memory_conditioned_features..cs6g|]\}}jr|nt||dfqS)r)rZabsr)rr! tpos_sign_mulr$r%r-s  rrrH)currcurr_posmemory memory_posnum_obj_ptr_tokens)+rr^rrbrr rrryvaluestrainingrmrangegetappendtorrerTminrUr[itemsziprstackrXrYr`tensorr rrexpand new_zerosreshaperepeat_interleaverarhrfrgcatr\)+r!ris_init_cond_framecurrent_vision_featscurrent_vision_pos_embedsr output_dict num_framesrrCHWrr to_cat_memoryto_cat_memory_pos_embed cond_outputsselected_cond_outputsunselected_cond_outputst_pos_and_prevsrt_post_relprev_frame_idxrprevfeats maskmem_encrUptr_cond_outputs pos_and_ptrst_diffrpos_list ptrs_listobj_ptrs t_diff_maxtpos_dimobj_pospix_feat_with_memrmemory_pos_embedr$)rr!rrr%$_prepare_memory_conditioned_featuresls                     z.SAM2Model._prepare_memory_conditioned_featurescCs|dd}|j}|d\}} |dddd|||| } |jr*|js*||}|jo.|} | r;|js;|dk} nt |} |j dkrJ| |j } |j dkrT| |j } |j | | dd} | d }| d }|jd ur|dk}|d|d |jd j|j7}||fS) zXEncode frame features and masks into a new memory representation for video segmentation.rrrrr?rT)skip_mask_sigmoidvision_featuresrN).NN)rr^rr rlr"_apply_non_overlapping_constraintsrkr9rrrirjr_rFr ra)r!rrpred_masks_high_resris_mask_from_ptsrrrrpix_featbinarize mask_for_mem maskmem_outrrrr$r$r%_encode_new_memorys2             zSAM2Model._encode_new_memoryc  Cs||d} t|dkrddt|dd|ddD} nd} |durE|jrE|dddd}|jd|jg|dR}||| |}n9|j|||dd|dd|dd|| | d }| durn|durj|duslJ| }|||}|j |||| |d }| || |fS) hPerform a single tracking step, updating object masks and memory features based on current frame inputs.)rrrcSs:g|]\}}|dddj|d|dg|RqS)rrr)rr r)r+r,sr$r$r%r-Ns(z)SAM2Model._track_step..Nrrr)rrrrrrrr)rrrrr) rr rnrr r^rr1_use_multimaskr)r!rrrrrrrrrrprev_sam_mask_logits current_outrr7 sam_outputsrr$r$r% _track_step<s@       zSAM2Model._track_stepc CsV|r!|jdkr!|}|j|||||dud\} } | |d<| |d<dSd|d<d|d<dS)z^Run memory encoder on predicted mask to encode it into a new memory feature for future frames.rN)rrr5rr6rr)rbr;) r!rrrrun_mem_encoderrrr@high_res_masks_for_mem_encrrr$r$r%_encode_memory_in_outputws    z"SAM2Model._encode_memory_in_outputc  Csx|||||||||| | | \} }}}|\}}}}}}}|| d<|| d<|| d<|js/|| d<||||| ||| | S)r< pred_masksr5rr)rBrrE)r!rrrrrrrrrrrCr?r@rArrrrrr$r$r% track_steps:   zSAM2Model.track_stepcCsF|durdn|dd}|jo"|p|jo"|j|ko |jkSS)zaDetermine whether to use multiple mask outputs in the SAM head based on configuration and inputs.Nrrr)rrorrrprq)r!rrnum_ptsr$r$r%r>szSAM2Model._use_multimaskcCsn|d}|dkr |S|j}tj|ddd}tj||ddddddf}||k}t||tj|dd}|S) z\Apply non-overlapping constraints to masks, keeping the highest scoring object per location.rrT)rkeepdimrNr)max)rrrrrrclamp)rF batch_sizer max_obj_indsbatch_obj_indskeepr$r$r%r4s z,SAM2Model._apply_non_overlapping_constraintscCs ||_dS)z Set binarize for VideoPredictor.N)rk)r!r8r$r$r% set_binarizes zSAM2Model.set_binarizecCs(|d|_||j_dd|D|j_dS)zCSet image size to make model compatible with different image sizes.rcSr'r(r$r*r$r$r%r-r.z'SAM2Model.set_imgsz..N)rtrr0r1r3r$r$r%r&s zSAM2Model.set_imgsz) r=r>r)r?rFFrFFFrrFFFrFFr)TFFFFFFFFFNF)NNNF)F)FTN)r5r6r7r8rr9r:boolrpropertyrr|rxrrrrrrr1r;rBrErGr> staticmethodr4rPr&r;r$r$r"r%r<js   !"$B 2 /  %,;% :  r<)typingrrtorch.nn.functionalr functionalr torch.nn.initrultralytics.nn.modulesrultralytics.utilsrblocksrdecodersr r encodersr r utilsr rrModulerr<r$r$r$r%s      N