o W h2@s`ddlZddlmZddlZddlmZmZddlmZGdddejZ Gdd d ejZ dS) N)Optional)Tensornn) RoPEAttentionc seZdZdZ      ddeded ed ed ed ef fd d Zdede edefddZ ddedede ede ededef ddZ   ddedede ede edede jf ddZ ZS) MemoryAttentionLayera Implements a memory attention layer with self-attention and cross-attention mechanisms for neural networks. This class combines self-attention, cross-attention, and feedforward components to process input tensors and generate memory-based attention outputs. Attributes: d_model (int): Dimensionality of the model. dim_feedforward (int): Dimensionality of the feedforward network. dropout_value (float): Dropout rate for regularization. self_attn (RoPEAttention): Self-attention mechanism using RoPE (Rotary Position Embedding). cross_attn_image (RoPEAttention): Cross-attention mechanism for image processing. linear1 (nn.Linear): First linear layer of the feedforward network. linear2 (nn.Linear): Second linear layer of the feedforward network. norm1 (nn.LayerNorm): Layer normalization for self-attention output. norm2 (nn.LayerNorm): Layer normalization for cross-attention output. norm3 (nn.LayerNorm): Layer normalization for feedforward network output. dropout1 (nn.Dropout): Dropout layer after self-attention. dropout2 (nn.Dropout): Dropout layer after cross-attention. dropout3 (nn.Dropout): Dropout layer after feedforward network. activation (nn.ReLU): Activation function for the feedforward network. pos_enc_at_attn (bool): Flag to add positional encoding at attention. pos_enc_at_cross_attn_queries (bool): Flag to add positional encoding to cross-attention queries. pos_enc_at_cross_attn_keys (bool): Flag to add positional encoding to cross-attention keys. Methods: forward: Performs the full memory attention operation on input tensors. _forward_sa: Performs self-attention on input tensor. _forward_ca: Performs cross-attention between target and memory tensors. Examples: >>> layer = MemoryAttentionLayer(d_model=256, dim_feedforward=2048, dropout=0.1) >>> tgt = torch.randn(1, 100, 256) >>> memory = torch.randn(1, 100, 64) >>> pos = torch.randn(1, 100, 256) >>> query_pos = torch.randn(1, 100, 256) >>> output = layer(tgt, memory, pos, query_pos) >>> print(output.shape) torch.Size([1, 100, 256]) 皙?FTd_modeldim_feedforwarddropoutpos_enc_at_attnpos_enc_at_cross_attn_keyspos_enc_at_cross_attn_queriescst||_||_||_tdddd|_tdddddd|_t |||_ t ||_ t |||_ t||_t||_t||_t ||_t ||_t ||_t|_||_||_||_dS)a{ Initialize a memory attention layer with self-attention, cross-attention, and feedforward components. Args: d_model (int): Dimensionality of the model. dim_feedforward (int): Dimensionality of the feedforward network. dropout (float): Dropout rate for regularization. pos_enc_at_attn (bool): Whether to add positional encoding at attention. pos_enc_at_cross_attn_keys (bool): Whether to add positional encoding to cross-attention keys. pos_enc_at_cross_attn_queries (bool): Whether to add positional encoding to cross-attention queries. rr) embedding_dim num_headsdownsample_rateT@) rope_k_repeatrrr kv_in_dimN)super__init__r r dropout_valuer self_attncross_attn_imagerLinearlinear1Dropoutr linear2 LayerNormnorm1norm2norm3dropout1dropout2dropout3ReLU activationrrr)selfr r r rrr __class__c/var/www/vscode/kcb/lib/python3.10/site-packages/ultralytics/models/sam/modules/memory_attention.pyr6s2          zMemoryAttentionLayer.__init__tgt query_posreturncCsB||}|jr ||n|}}|j|||d}|||}|S)z^Perform self-attention on input tensor using positional encoding and RoPE attention mechanism.)v)r!rrr$)r)r.r/tgt2qkr,r,r- _forward_sajs z MemoryAttentionLayer._forward_sarmemoryposnum_k_exclude_ropecCsti}|dkrt|jtsJd|i}||}|jd|jr!||n||jr)||n||d|}|||}|S)zXPerform cross-attention between target and memory tensors using RoPEAttention mechanism.rr8)r3r4r1Nr,) isinstancerrr"rrr%)r)r.r6r/r7r8kwdsr2r,r,r- _forward_cars  z MemoryAttentionLayer._forward_caNc CsV|||}||||||}||}|||||}|||}|S)z^Process input tensors through self-attention, cross-attention, and feedforward network layers.)r5r;r#rr r(rr&)r)r.r6r7r/r8r2r,r,r-forwards  zMemoryAttentionLayer.forward)rr r FTF)rNNr)__name__ __module__ __qualname____doc__intfloatboolrrrr5r;torchr< __classcell__r,r,r*r-r sh+4 rc steZdZdZ ddededejdedef fdd Z dd e j d e j de e de e dede j f ddZ Z S)MemoryAttentiona Memory attention module for processing sequential data with self and cross-attention mechanisms. This class implements a multi-layer attention mechanism that combines self-attention and cross-attention for processing sequential data, particularly useful in transformer-like architectures. Attributes: d_model (int): The dimension of the model's hidden state. layers (nn.ModuleList): A list of MemoryAttentionLayer modules. num_layers (int): The number of attention layers. norm (nn.LayerNorm): Layer normalization applied to the output. pos_enc_at_input (bool): Whether to apply positional encoding at the input. batch_first (bool): Whether the input tensors are in batch-first format. Methods: forward: Processes input tensors through the attention layers. Examples: >>> d_model = 256 >>> layer = MemoryAttentionLayer(d_model) >>> attention = MemoryAttention(d_model, pos_enc_at_input=True, layer=layer, num_layers=3) >>> curr = torch.randn(10, 32, d_model) # (seq_len, batch_size, d_model) >>> memory = torch.randn(20, 32, d_model) # (mem_len, batch_size, d_model) >>> curr_pos = torch.randn(10, 32, d_model) >>> memory_pos = torch.randn(20, 32, d_model) >>> output = attention(curr, memory, curr_pos, memory_pos) >>> print(output.shape) torch.Size([10, 32, 256]) Tr pos_enc_at_inputlayer num_layers batch_firstcsPt||_tfddt|D|_||_t||_ ||_ ||_ dS)aQ Initialize MemoryAttention with specified layers and normalization for sequential data processing. This class implements a multi-layer attention mechanism that combines self-attention and cross-attention for processing sequential data, particularly useful in transformer-like architectures. Args: d_model (int): The dimension of the model's hidden state. pos_enc_at_input (bool): Whether to apply positional encoding at the input. layer (nn.Module): The attention layer to be used in the module. num_layers (int): The number of attention layers. batch_first (bool): Whether the input tensors are in batch-first format. Examples: >>> d_model = 256 >>> layer = MemoryAttentionLayer(d_model) >>> attention = MemoryAttention(d_model, pos_enc_at_input=True, layer=layer, num_layers=3) >>> curr = torch.randn(10, 32, d_model) # (seq_len, batch_size, d_model) >>> memory = torch.randn(20, 32, d_model) # (mem_len, batch_size, d_model) >>> curr_pos = torch.randn(10, 32, d_model) >>> memory_pos = torch.randn(20, 32, d_model) >>> output = attention(curr, memory, curr_pos, memory_pos) >>> print(output.shape) torch.Size([10, 32, 256]) csg|]}tqSr,)copydeepcopy).0_rIr,r- sz,MemoryAttention.__init__..N) rrr r ModuleListrangelayersrJr normrHrK)r)r rHrIrJrKr*rPr-rs !  zMemoryAttention.__init__Nrcurrr6curr_pos memory_posnum_obj_ptr_tokensr0c Cs&t|tr't|ts Jt|t|krdksJJ|d|d}}|jd|jdks5Jd|}|jrD|durD|d|}|jr_|dd}|dd}|dd}|dd}|jD]}i}t|jt rpd|i}|d||||d|}qb| |} |jr| dd} |dd}| S) aa Process inputs through attention layers, applying self and cross-attention with positional encoding. Args: curr (torch.Tensor): Self-attention input tensor, representing the current state. memory (torch.Tensor): Cross-attention input tensor, representing memory information. curr_pos (Optional[Tensor]): Positional encoding for self-attention inputs. memory_pos (Optional[Tensor]): Positional encoding for cross-attention inputs. num_obj_ptr_tokens (int): Number of object pointer tokens to exclude from rotary position embedding. Returns: (torch.Tensor): Processed output tensor after applying attention layers and normalization. Examples: >>> d_model = 256 >>> layer = MemoryAttentionLayer(d_model) >>> attention = MemoryAttention(d_model, pos_enc_at_input=True, layer=layer, num_layers=3) >>> curr = torch.randn(10, 32, d_model) # (seq_len, batch_size, d_model) >>> memory = torch.randn(20, 32, d_model) # (mem_len, batch_size, d_model) >>> curr_pos = torch.randn(10, 32, d_model) >>> memory_pos = torch.randn(20, 32, d_model) >>> output = attention(curr, memory, curr_pos, memory_pos) >>> print(output.shape) torch.Size([10, 32, 256]) rrz/Batch size must be the same for curr and memoryNr r8)r.r6r7r/r,) r9listlenshaperHrK transposerTrrrU) r)rVr6rWrXrYoutputrIr: normed_outputr,r,r-r<s< !$          zMemoryAttention.forward)Tr=)r>r?r@rArBrDrModulerrErrr<rFr,r,r*r-rGs<$-rG) rLtypingrrErrblocksrr`rrGr,r,r,r-s