o W h6d@sdddlmZmZmZmZddlZddlmZddlmZm Z Gdddej Z Gdddej Z dS) )ListOptionalTupleTypeN)nn)MLP LayerNorm2dcseZdZdZdejddfdedejdedeejded ed d ffd d Z de j de j de j de j de d e e j e j ff ddZde j de j de j de j d e e j e j ff ddZZS) MaskDecodera Decoder module for generating masks and their associated quality scores using a transformer architecture. This class predicts masks given image and prompt embeddings, utilizing a transformer to process the inputs and generate mask predictions along with their quality scores. Attributes: transformer_dim (int): Channel dimension for the transformer module. transformer (nn.Module): Transformer module used for mask prediction. num_multimask_outputs (int): Number of masks to predict for disambiguating masks. iou_token (nn.Embedding): Embedding for the IoU token. num_mask_tokens (int): Number of mask tokens. mask_tokens (nn.Embedding): Embedding for the mask tokens. output_upscaling (nn.Sequential): Neural network sequence for upscaling the output. output_hypernetworks_mlps (nn.ModuleList): Hypernetwork MLPs for generating masks. iou_prediction_head (nn.Module): MLP for predicting mask quality. Methods: forward: Predicts masks given image and prompt embeddings. predict_masks: Internal method for mask prediction. Examples: >>> decoder = MaskDecoder(transformer_dim=256, transformer=transformer_module) >>> masks, iou_pred = decoder( ... image_embeddings, image_pe, sparse_prompt_embeddings, dense_prompt_embeddings, multimask_output=True ... ) >>> print(f"Predicted masks shape: {masks.shape}, IoU predictions shape: {iou_pred.shape}") transformer_dim transformernum_multimask_outputs activationiou_head_depthiou_head_hidden_dimreturnNc st|_||_||_td|_|d|_t|j|_ t tj ddddt d|tj ddddd||_ tfddt|jD|_t||j||_dS) a Initialize the MaskDecoder module for generating masks and their associated quality scores. Args: transformer_dim (int): Channel dimension for the transformer module. transformer (nn.Module): Transformer module used for mask prediction. num_multimask_outputs (int): Number of masks to predict for disambiguating masks. activation (Type[nn.Module]): Type of activation to use when upscaling masks. iou_head_depth (int): Depth of the MLP used to predict mask quality. iou_head_hidden_dim (int): Hidden dimension of the MLP used to predict mask quality. Examples: >>> transformer = nn.TransformerEncoder(nn.TransformerEncoderLayer(d_model=256, nhead=8), num_layers=6) >>> decoder = MaskDecoder(transformer_dim=256, transformer=transformer) >>> print(decoder)  kernel_sizestridecg|] }tddqSrr r.0_r [/var/www/vscode/kcb/lib/python3.10/site-packages/ultralytics/models/sam/modules/decoders.py Tz(MaskDecoder.__init__..N)super__init__r r rr Embedding iou_tokennum_mask_tokens mask_tokens SequentialConvTranspose2droutput_upscaling ModuleListrangeoutput_hypernetworks_mlpsriou_prediction_head)selfr r rrrr __class__r r"r&)s$   zMaskDecoder.__init__image_embeddingsimage_pesparse_prompt_embeddingsdense_prompt_embeddingsmultimask_outputc Csb|j||||d\}}|rtddntdd}|dd|ddddf}|dd|f}||fS)a Predict masks given image and prompt embeddings. Args: image_embeddings (torch.Tensor): Embeddings from the image encoder. image_pe (torch.Tensor): Positional encoding with the shape of image_embeddings. sparse_prompt_embeddings (torch.Tensor): Embeddings of the points and boxes. dense_prompt_embeddings (torch.Tensor): Embeddings of the mask inputs. multimask_output (bool): Whether to return multiple masks or a single mask. Returns: masks (torch.Tensor): Batched predicted masks. iou_pred (torch.Tensor): Batched predictions of mask quality. Examples: >>> decoder = MaskDecoder(transformer_dim=256, transformer=transformer_module) >>> image_emb = torch.rand(1, 256, 64, 64) >>> image_pe = torch.rand(1, 256, 64, 64) >>> sparse_emb = torch.rand(1, 2, 256) >>> dense_emb = torch.rand(1, 256, 64, 64) >>> masks, iou_pred = decoder(image_emb, image_pe, sparse_emb, dense_emb, multimask_output=True) >>> print(f"Masks shape: {masks.shape}, IoU predictions shape: {iou_pred.shape}") )r5r6r7r8rNr) predict_masksslice) r2r5r6r7r8r9masksiou_pred mask_slicer!r!r"forwardYs zMaskDecoder.forwardcs\tjjjjjgdd}|d|jddd}tj||fdd}tj||jddd}||}tj||jddd}|j\} } } }  |||\} }| dddddf}| ddddj ddf| dd | | | | } |}fddtj D}tj|dd}|j\} } } } || | | | |  | d| | }|}||fS) z`Predict masks and quality scores using image and prompt embeddings via transformer architecture.rdimrNrc,g|]}j|dd|ddfqSNr0rimask_tokens_outr2r!r"r# z-MaskDecoder.predict_masks..)torchcatr(weightr* unsqueezeexpandshaperepeat_interleaver r) transposeviewr-r/stackr1)r2r5r6r7r8 output_tokenstokenssrcpos_srcbchwhs iou_token_outupscaled_embedding hyper_in_listhyper_inr<r=r!rHr"r:s(    " zMaskDecoder.predict_masks)__name__ __module__ __qualname____doc__rGELUintModulerr&rKTensorboolrr?r: __classcell__r!r!r3r"r sZ!0 .r cs&eZdZdZdejddddddddddf dedejd ed eejd ed ed e de de de ddffdd Z d$de j de j de j de j de de de ee j dee j e j e j e j ffddZ d$de j de j de j de j de de ee j dee j e j e j e j ffddZd d!Zd"d#ZZS)%SAM2MaskDecodera Transformer-based decoder for predicting instance segmentation masks from image and prompt embeddings. This class extends the functionality of the MaskDecoder, incorporating additional features such as high-resolution feature processing, dynamic multimask output, and object score prediction. Attributes: transformer_dim (int): Channel dimension of the transformer. transformer (nn.Module): Transformer used to predict masks. num_multimask_outputs (int): Number of masks to predict when disambiguating masks. iou_token (nn.Embedding): Embedding for IOU token. num_mask_tokens (int): Total number of mask tokens. mask_tokens (nn.Embedding): Embedding for mask tokens. pred_obj_scores (bool): Whether to predict object scores. obj_score_token (nn.Embedding): Embedding for object score token. use_multimask_token_for_obj_ptr (bool): Whether to use multimask token for object pointer. output_upscaling (nn.Sequential): Upscaling layers for output. use_high_res_features (bool): Whether to use high-resolution features. conv_s0 (nn.Conv2d): Convolutional layer for high-resolution features (s0). conv_s1 (nn.Conv2d): Convolutional layer for high-resolution features (s1). output_hypernetworks_mlps (nn.ModuleList): List of MLPs for output hypernetworks. iou_prediction_head (MLP): MLP for IOU prediction. pred_obj_score_head (nn.Linear | MLP): Linear layer or MLP for object score prediction. dynamic_multimask_via_stability (bool): Whether to use dynamic multimask via stability. dynamic_multimask_stability_delta (float): Delta value for dynamic multimask stability. dynamic_multimask_stability_thresh (float): Threshold for dynamic multimask stability. Methods: forward: Predicts masks given image and prompt embeddings. predict_masks: Predicts instance segmentation masks from image and prompt embeddings. _get_stability_scores: Computes mask stability scores based on IoU between thresholds. _dynamic_multimask_via_stability: Dynamically selects the most stable mask output. Examples: >>> image_embeddings = torch.rand(1, 256, 64, 64) >>> image_pe = torch.rand(1, 256, 64, 64) >>> sparse_prompt_embeddings = torch.rand(1, 2, 256) >>> dense_prompt_embeddings = torch.rand(1, 256, 64, 64) >>> decoder = SAM2MaskDecoder(256, transformer) >>> masks, iou_pred, sam_tokens_out, obj_score_logits = decoder.forward( ... image_embeddings, image_pe, sparse_prompt_embeddings, dense_prompt_embeddings, True, False ... ) r r Fg?g\(\?r r rrrruse_high_res_featurespred_obj_scorespred_obj_scores_mlpuse_multimask_token_for_obj_ptrrNc sZt|_||_||_td|_|d|_t|j|_ | |_ |j r/td|_ ||_ t tjddddtd|tjddddd||_||_|rstjdddd|_tjdddd|_tfddt|jD|_t||j||d|_|j rtd|_| rtdd |_| |_| |_| |_d S) a Initialize the SAM2MaskDecoder module for predicting instance segmentation masks. This decoder extends the functionality of MaskDecoder, incorporating additional features such as high-resolution feature processing, dynamic multimask output, and object score prediction. Args: transformer_dim (int): Channel dimension of the transformer. transformer (nn.Module): Transformer used to predict masks. num_multimask_outputs (int): Number of masks to predict when disambiguating masks. activation (Type[nn.Module]): Type of activation to use when upscaling masks. iou_head_depth (int): Depth of the MLP used to predict mask quality. iou_head_hidden_dim (int): Hidden dimension of the MLP used to predict mask quality. use_high_res_features (bool): Whether to use high-resolution features. iou_prediction_use_sigmoid (bool): Whether to use sigmoid for IOU prediction. dynamic_multimask_via_stability (bool): Whether to use dynamic multimask via stability. dynamic_multimask_stability_delta (float): Delta value for dynamic multimask stability. dynamic_multimask_stability_thresh (float): Threshold for dynamic multimask stability. pred_obj_scores (bool): Whether to predict object scores. pred_obj_scores_mlp (bool): Whether to use MLP for object score prediction. use_multimask_token_for_obj_ptr (bool): Whether to use multimask token for object pointer. Examples: >>> transformer = nn.TransformerEncoder(nn.TransformerEncoderLayer(d_model=256, nhead=8), num_layers=6) >>> decoder = SAM2MaskDecoder(transformer_dim=256, transformer=transformer) >>> print(decoder) rrrrrcrrrrr r!r"r#$r$z,SAM2MaskDecoder.__init__..)sigmoidr N)r%r&r r rrr'r(r)r*rnobj_score_tokenrpr+r,rr-rmConv2dconv_s0conv_s1r.r/r0rr1Linearpred_obj_score_headdynamic_multimask_via_stability!dynamic_multimask_stability_delta"dynamic_multimask_stability_thresh)r2r r rrrrrmiou_prediction_use_sigmoidrxryrzrnrorpr3r r"r&sN ,   zSAM2MaskDecoder.__init__r5r6r7r8r9 repeat_imagehigh_res_featuresc Cs|j||||||d\}} } } |r,|ddddddddf}| ddddf} n)|jr;|js;||| \}} n|ddddddddf}| ddddf} |re|jre| ddddf} n | ddddf} || | | fS)a Predict masks given image and prompt embeddings. Args: image_embeddings (torch.Tensor): Embeddings from the image encoder with shape (B, C, H, W). image_pe (torch.Tensor): Positional encoding with the shape of image_embeddings (B, C, H, W). sparse_prompt_embeddings (torch.Tensor): Embeddings of the points and boxes with shape (B, N, C). dense_prompt_embeddings (torch.Tensor): Embeddings of the mask inputs with shape (B, C, H, W). multimask_output (bool): Whether to return multiple masks or a single mask. repeat_image (bool): Flag to repeat the image embeddings. high_res_features (List[torch.Tensor] | None): Optional high-resolution features. Returns: masks (torch.Tensor): Batched predicted masks with shape (B, N, H, W). iou_pred (torch.Tensor): Batched predictions of mask quality with shape (B, N). sam_tokens_out (torch.Tensor): Batched SAM token for mask output with shape (B, N, C). object_score_logits (torch.Tensor): Batched object score logits with shape (B, 1). Examples: >>> image_embeddings = torch.rand(1, 256, 64, 64) >>> image_pe = torch.rand(1, 256, 64, 64) >>> sparse_prompt_embeddings = torch.rand(1, 2, 256) >>> dense_prompt_embeddings = torch.rand(1, 256, 64, 64) >>> decoder = SAM2MaskDecoder(256, transformer) >>> masks, iou_pred, sam_tokens_out, obj_score_logits = decoder.forward( ... image_embeddings, image_pe, sparse_prompt_embeddings, dense_prompt_embeddings, True, False ... ) )r5r6r7r8r|r}Nrr)r:rxtraining _dynamic_multimask_via_stabilityrp) r2r5r6r7r8r9r|r}r<r=rIobject_score_logitssam_tokens_outr!r!r"r?9s&&     zSAM2MaskDecoder.forwardcs^d}jrtjjjjjjjgdd}d}n tjjjjjgdd}|d| ddd}tj||fdd} |rItj || j ddd} n|j d| j dksUJ|} | |} | ddksfJdtj || j ddd} | j \} } }} | | | \}} |dd|ddf}|dd|d|dj ddf| dd| | ||} js| }nj\}}}}}|\}}|||| |}||||}fdd tj D}tj|dd}|j \} } }}||| | ||| d||}|}jr|dksJ|dddddf}n d ||j dd}|||fS) zYPredict instance segmentation masks from image and prompt embeddings using a transformer.rr@rrBz@image_pe should have size 1 in batch dim (from `get_dense_pe()`)NrcrCrDrErFrHr!r"r#rJz1SAM2MaskDecoder.predict_masks..g$@)rnrKrLrrrMr(r*rNrOsizerQrPr r)rRrSrmr-r/rTr1rwnew_ones)r2r5r6r7r8r|r}srUrVrWrXrYrZr[r\r]r^r_dc1ln1act1dc2act2feat_s0feat_s1r`rar<r=rr!rHr"r:sV (  "  zSAM2MaskDecoder.predict_maskscCsT|d}|j}tj||kdd}tj|| kdd}t|dk||dS)zNCompute mask stability scores based on IoU between upper and lower thresholds.rBr@rg?)flattenryrKsumfloatwhere)r2 mask_logitsstability_deltaarea_iarea_ur!r!r"_get_stability_scoress z%SAM2MaskDecoder._get_stability_scorescCs|ddddddddf}|ddddf}tj|dd}tj|d|jd}|||f}|d}|||f}|d}|ddddddddf} |ddddf} || } | |jk} t| d | | |} t| | | |}| |fS)a Dynamically select the most stable mask output based on stability scores and IoU predictions. This method is used when outputting a single mask. If the stability score from the current single-mask output (based on output token 0) falls below a threshold, it instead selects from multi-mask outputs (based on output tokens 1-3) the mask with the highest predicted IoU score. This ensures a valid mask for both clicking and tracking scenarios. Args: all_mask_logits (torch.Tensor): Logits for all predicted masks, shape (B, N, H, W) where B is batch size, N is number of masks (typically 4), and H, W are mask dimensions. all_iou_scores (torch.Tensor): Predicted IoU scores for all masks, shape (B, N). Returns: mask_logits_out (torch.Tensor): Selected mask logits, shape (B, 1, H, W). iou_scores_out (torch.Tensor): Selected IoU scores, shape (B, 1). Examples: >>> decoder = SAM2MaskDecoder(...) >>> all_mask_logits = torch.rand(2, 4, 256, 256) # 2 images, 4 masks each >>> all_iou_scores = torch.rand(2, 4) >>> mask_logits, iou_scores = decoder._dynamic_multimask_via_stability(all_mask_logits, all_iou_scores) >>> print(mask_logits.shape, iou_scores.shape) torch.Size([2, 1, 256, 256]) torch.Size([2, 1]) NrrBr@r)device).NN) rKargmaxarangerrrNrrzr expand_as)r2all_mask_logitsall_iou_scoresmultimask_logitsmultimask_iou_scoresbest_scores_inds batch_indsbest_multimask_logitsbest_multimask_iou_scoressinglemask_logitssinglemask_iou_scoresstability_scores is_stablemask_logits_outiou_scores_outr!r!r"rs.         z0SAM2MaskDecoder._dynamic_multimask_via_stabilityrD)rbrcrdrerrfrgrhrrjr&rKrirrrr?r:rrrkr!r!r3r"rls0 e  M  Grl) typingrrrrrKrultralytics.nn.modulesrrrhr rlr!r!r!r"s %