o W hR9@svddlZddlmZmZddlZddlmZmZddlmZGdddej Z Gdddej Z Gd d d ej Z dS) N)TupleType)Tensornn)MLPBlockcsneZdZdZejdfdededededeejded d ffd d Z d e de de d e e e ffddZ Z S)TwoWayTransformera A Two-Way Transformer module for simultaneous attention to image and query points. This class implements a specialized transformer decoder that attends to an input image using queries with supplied positional embeddings. It's useful for tasks like object detection, image segmentation, and point cloud processing. Attributes: depth (int): Number of layers in the transformer. embedding_dim (int): Channel dimension for input embeddings. num_heads (int): Number of heads for multihead attention. mlp_dim (int): Internal channel dimension for the MLP block. layers (nn.ModuleList): List of TwoWayAttentionBlock layers composing the transformer. final_attn_token_to_image (Attention): Final attention layer from queries to image. norm_final_attn (nn.LayerNorm): Layer normalization applied to final queries. Methods: forward: Processes image and point embeddings through the transformer. Examples: >>> transformer = TwoWayTransformer(depth=6, embedding_dim=256, num_heads=8, mlp_dim=2048) >>> image_embedding = torch.randn(1, 256, 32, 32) >>> image_pe = torch.randn(1, 256, 32, 32) >>> point_embedding = torch.randn(1, 100, 256) >>> output_queries, output_image = transformer(image_embedding, image_pe, point_embedding) >>> print(output_queries.shape, output_image.shape) depth embedding_dim num_headsmlp_dim activationattention_downsample_ratereturnNc szt||_||_||_||_t|_t |D]}|j t ||||||dkdqt |||d|_ t||_dS)aW Initialize a Two-Way Transformer for simultaneous attention to image and query points. Args: depth (int): Number of layers in the transformer. embedding_dim (int): Channel dimension for input embeddings. num_heads (int): Number of heads for multihead attention. Must divide embedding_dim. mlp_dim (int): Internal channel dimension for the MLP block. activation (Type[nn.Module]): Activation function to use in the MLP block. attention_downsample_rate (int): Downsampling rate for attention mechanism. r)r r r r rskip_first_layer_pedownsample_rateN)super__init__r r r r r ModuleListlayersrangeappendTwoWayAttentionBlock Attentionfinal_attn_token_to_image LayerNormnorm_final_attn)selfr r r r r ri __class__^/var/www/vscode/kcb/lib/python3.10/site-packages/ultralytics/models/sam/modules/transformer.pyr)s&    zTwoWayTransformer.__init__image_embeddingimage_pepoint_embeddingc Cs|dddd}|dddd}|}|}|jD] }|||||d\}}q||}||}|j|||d} || }||}||fS)an Process image and point embeddings through the Two-Way Transformer. Args: image_embedding (Tensor): Image to attend to, with shape (B, embedding_dim, H, W). image_pe (Tensor): Positional encoding to add to the image, with same shape as image_embedding. point_embedding (Tensor): Embedding to add to query points, with shape (B, N_points, embedding_dim). Returns: queries (Tensor): Processed point embeddings with shape (B, N_points, embedding_dim). keys (Tensor): Processed image embeddings with shape (B, H*W, embedding_dim). rr)querieskeysquery_pekey_peqkv)flattenpermuterrr) rr$r%r&r(r)layerr-r.attn_outr"r"r#forwardSs"   zTwoWayTransformer.forward)__name__ __module__ __qualname____doc__rReLUintrModulerrrr4 __classcell__r"r"r r#r s8"* rcsveZdZdZdejddfdedededeejd ed e d d ffd d Z de de de de d e e e ff ddZ ZS)raI A two-way attention block for simultaneous attention to image and query points. This class implements a specialized transformer block with four main layers: self-attention on sparse inputs, cross-attention of sparse inputs to dense inputs, MLP block on sparse inputs, and cross-attention of dense inputs to sparse inputs. Attributes: self_attn (Attention): Self-attention layer for queries. norm1 (nn.LayerNorm): Layer normalization after self-attention. cross_attn_token_to_image (Attention): Cross-attention layer from queries to keys. norm2 (nn.LayerNorm): Layer normalization after token-to-image attention. mlp (MLPBlock): MLP block for transforming query embeddings. norm3 (nn.LayerNorm): Layer normalization after MLP block. norm4 (nn.LayerNorm): Layer normalization after image-to-token attention. cross_attn_image_to_token (Attention): Cross-attention layer from keys to queries. skip_first_layer_pe (bool): Whether to skip positional encoding in the first layer. Methods: forward: Applies self-attention and cross-attention to queries and keys. Examples: >>> embedding_dim, num_heads = 256, 8 >>> block = TwoWayAttentionBlock(embedding_dim, num_heads) >>> queries = torch.randn(1, 100, embedding_dim) >>> keys = torch.randn(1, 1000, embedding_dim) >>> query_pe = torch.randn(1, 100, embedding_dim) >>> key_pe = torch.randn(1, 1000, embedding_dim) >>> processed_queries, processed_keys = block(queries, keys, query_pe, key_pe) irFr r r r rrrNcs~tt|||_t||_t|||d|_t||_t ||||_ t||_ t||_ t|||d|_ ||_dS)ah Initialize a TwoWayAttentionBlock for simultaneous attention to image and query points. This block implements a specialized transformer layer with four main components: self-attention on sparse inputs, cross-attention of sparse inputs to dense inputs, MLP block on sparse inputs, and cross-attention of dense inputs to sparse inputs. Args: embedding_dim (int): Channel dimension of the embeddings. num_heads (int): Number of attention heads in the attention layers. mlp_dim (int): Hidden dimension of the MLP block. activation (Type[nn.Module]): Activation function for the MLP block. attention_downsample_rate (int): Downsampling rate for the attention mechanism. skip_first_layer_pe (bool): Whether to skip positional encoding in the first layer. rN)rrr self_attnrrnorm1cross_attn_token_to_imagenorm2rmlpnorm3norm4cross_attn_image_to_tokenr)rr r r r rrr r"r#rs       zTwoWayAttentionBlock.__init__r(r)r*r+c Cs|jr |j|||d}n||}|j|||d}||}||}||}||}|j|||d}||}||}||}||}||}||}||}|j|||d}||}||}||fS)a Apply two-way attention to process query and key embeddings in a transformer block. Args: queries (Tensor): Query embeddings with shape (B, N_queries, embedding_dim). keys (Tensor): Key embeddings with shape (B, N_keys, embedding_dim). query_pe (Tensor): Positional encodings for queries with same shape as queries. key_pe (Tensor): Positional encodings for keys with same shape as keys. Returns: queries (Tensor): Processed query embeddings with shape (B, N_queries, embedding_dim). keys (Tensor): Processed key embeddings with shape (B, N_keys, embedding_dim). r,) rr=r>r?r@rArBrDrC) rr(r)r*r+r-r3r.mlp_outr"r"r#r4s(     zTwoWayAttentionBlock.forward)r5r6r7r8rr9r:rr;boolrrrr4r<r"r"r r#rs,#.'rc seZdZdZ  ddededededdf fd d Zed ededefd d Zed edefddZ dedededefddZ Z S)ra An attention layer with downscaling capability for embedding size after projection. This class implements a multi-head attention mechanism with the option to downsample the internal dimension of queries, keys, and values. Attributes: embedding_dim (int): Dimensionality of input embeddings. kv_in_dim (int): Dimensionality of key and value inputs. internal_dim (int): Internal dimension after downsampling. num_heads (int): Number of attention heads. q_proj (nn.Linear): Linear projection for queries. k_proj (nn.Linear): Linear projection for keys. v_proj (nn.Linear): Linear projection for values. out_proj (nn.Linear): Linear projection for output. Methods: _separate_heads: Separates input tensor into attention heads. _recombine_heads: Recombines separated attention heads. forward: Computes attention output for given query, key, and value tensors. Examples: >>> attn = Attention(embedding_dim=256, num_heads=8, downsample_rate=2) >>> q = torch.randn(1, 100, 256) >>> k = v = torch.randn(1, 50, 256) >>> output = attn(q, k, v) >>> print(output.shape) torch.Size([1, 100, 256]) r'Nr r r kv_in_dimrcst||_|dur|n||_|||_||_|j|dks$Jdt||j|_t|j|j|_ t|j|j|_ t|j||_ dS)a+ Initialize the Attention module with specified dimensions and settings. Args: embedding_dim (int): Dimensionality of input embeddings. num_heads (int): Number of attention heads. downsample_rate (int): Factor by which internal dimensions are downsampled. kv_in_dim (int | None): Dimensionality of key and value inputs. If None, uses embedding_dim. Raises: AssertionError: If num_heads does not evenly divide the internal dim (embedding_dim / downsample_rate). Nrz$num_heads must divide embedding_dim.) rrr rG internal_dimr rLinearq_projk_projv_projout_proj)rr r rrGr r"r#rs  zAttention.__init__xcCs,|j\}}}||||||}|ddS)zGSeparate the input tensor into the specified number of attention heads.r'r)shapereshape transpose)rNr bncr"r"r#_separate_heads2s  zAttention._separate_headscCs,|j\}}}}|dd}|||||S)z9Recombine separated attention heads into a single tensor.r'r)rOrQrP)rNrRn_headsn_tokens c_per_headr"r"r#_recombine_heads9s zAttention._recombine_headsr-r.r/cCs||}||}||}|||j}|||j}|||j}|j\}}}}||dddd}|t|}t j |dd}||}| |}| |S)a Apply multi-head attention to query, key, and value tensors with optional downsampling. Args: q (Tensor): Query tensor with shape (B, N_q, embedding_dim). k (Tensor): Key tensor with shape (B, N_k, embedding_dim). v (Tensor): Value tensor with shape (B, N_k, embedding_dim). Returns: (Tensor): Output tensor after attention with shape (B, N_q, embedding_dim). rr'r)dim) rJrKrLrUr rOr1mathsqrttorchsoftmaxrYrM)rr-r.r/_rXattnoutr"r"r#r4@s    zAttention.forward)r'N) r5r6r7r8r:r staticmethodrrUrYr4r<r"r"r r#rs(""r) r]typingrrr_rrultralytics.nn.modulesrr;rrrr"r"r"r#s tt