o Wñ hCnã@sdZddlZddlZddlmZddlmmZddlm Z m Z ddl m Z ddl mZmZmZdZGdd „d ejƒZGd d „d eƒZGd d „d ejƒZGdd„dejƒZGdd„dejƒZGdd„dejƒZGdd„dejƒZGdd„dejƒZGdd„dejƒZGdd„dejƒZdS)zTransformer modules.éN)Ú constant_Úxavier_uniform_é)ÚConv)Ú _get_clonesÚinverse_sigmoidÚ#multi_scale_deformable_attn_pytorch) ÚTransformerEncoderLayerÚTransformerLayerÚTransformerBlockÚMLPBlockÚ LayerNorm2dÚAIFIÚDeformableTransformerDecoderÚ!DeformableTransformerDecoderLayerÚ MSDeformAttnÚMLPcs\eZdZdZddde ¡df‡fdd„ Zedd d „ƒZdd d „Z dd d„Z ddd„Z ‡Z S)r a Defines a single layer of the transformer encoder. Attributes: ma (nn.MultiheadAttention): Multi-head attention module. fc1 (nn.Linear): First linear layer in the feedforward network. fc2 (nn.Linear): Second linear layer in the feedforward network. norm1 (nn.LayerNorm): Layer normalization after attention. norm2 (nn.LayerNorm): Layer normalization after feedforward network. dropout (nn.Dropout): Dropout layer for the feedforward network. dropout1 (nn.Dropout): Dropout layer after attention. dropout2 (nn.Dropout): Dropout layer after feedforward network. act (nn.Module): Activation function. normalize_before (bool): Whether to apply normalization before attention and feedforward. ééçFcsžtƒ ¡ddlm}|stdƒ‚tj|||dd|_t ||¡|_ t ||¡|_ t  |¡|_ t  |¡|_ t |¡|_t |¡|_t |¡|_||_||_dS)aÎ Initialize the TransformerEncoderLayer with specified parameters. Args: c1 (int): Input dimension. cm (int): Hidden dimension in the feedforward network. num_heads (int): Number of attention heads. dropout (float): Dropout probability. act (nn.Module): Activation function. normalize_before (bool): Whether to apply normalization before attention and feedforward. é)Ú TORCH_1_9z]TransformerEncoderLayer() requires torch>=1.9 to use nn.MultiheadAttention(batch_first=True).T)ÚdropoutÚ batch_firstN)ÚsuperÚ__init__Úutils.torch_utilsrÚModuleNotFoundErrorÚnnÚMultiheadAttentionÚmaÚLinearÚfc1Úfc2Ú LayerNormÚnorm1Únorm2ÚDropoutrÚdropout1Údropout2ÚactÚnormalize_before)ÚselfÚc1ÚcmÚ num_headsrr*r+r©Ú __class__©úV/var/www/vscode/kcb/lib/python3.10/site-packages/ultralytics/nn/modules/transformer.pyr-s ÿ      z TransformerEncoderLayer.__init__NcCó|dur|S||S)z2Add position embeddings to the tensor if provided.Nr2©ÚtensorÚposr2r2r3Úwith_pos_embedNóz&TransformerEncoderLayer.with_pos_embedc Cst| ||¡}}|j|||||dd}|| |¡}| |¡}| | | | |¡¡¡¡}|| |¡}|  |¡S)a¾ Perform forward pass with post-normalization. Args: src (torch.Tensor): Input tensor. src_mask (torch.Tensor, optional): Mask for the src sequence. src_key_padding_mask (torch.Tensor, optional): Mask for the src keys per batch. pos (torch.Tensor, optional): Positional encoding. Returns: (torch.Tensor): Output tensor after attention and feedforward. ©ÚvalueÚ attn_maskÚkey_padding_maskr) r8r r(r%r#rr*r"r)r&)r,ÚsrcÚsrc_maskÚsrc_key_padding_maskr7ÚqÚkÚsrc2r2r2r3Ú forward_postSs   z$TransformerEncoderLayer.forward_postc Cst| |¡}| ||¡}}|j|||||dd}|| |¡}| |¡}| | | | |¡¡¡¡}||  |¡S)a½ Perform forward pass with pre-normalization. Args: src (torch.Tensor): Input tensor. src_mask (torch.Tensor, optional): Mask for the src sequence. src_key_padding_mask (torch.Tensor, optional): Mask for the src keys per batch. pos (torch.Tensor, optional): Positional encoding. Returns: (torch.Tensor): Output tensor after attention and feedforward. r:r) r%r8r r(r&r#rr*r"r))r,r>r?r@r7rCrArBr2r2r3Ú forward_prehs  z#TransformerEncoderLayer.forward_precCs&|jr | ||||¡S| ||||¡S)aÉ Forward propagates the input through the encoder module. Args: src (torch.Tensor): Input tensor. src_mask (torch.Tensor, optional): Mask for the src sequence. src_key_padding_mask (torch.Tensor, optional): Mask for the src keys per batch. pos (torch.Tensor, optional): Positional encoding. Returns: (torch.Tensor): Output tensor after transformer encoder layer. )r+rErD)r,r>r?r@r7r2r2r3Úforward}s zTransformerEncoderLayer.forward©N©NNN) Ú__name__Ú __module__Ú __qualname__Ú__doc__rÚGELUrÚ staticmethodr8rDrErFÚ __classcell__r2r2r0r3r s!   r csJeZdZdZddde ¡df‡fdd„ Z‡fdd „Zedd d „ƒZ ‡Z S)rz“ Defines the AIFI transformer layer. This class extends TransformerEncoderLayer to work with 2D data by adding positional embeddings. rrrFcstƒ ||||||¡dS)aÄ Initialize the AIFI instance with specified parameters. Args: c1 (int): Input dimension. cm (int): Hidden dimension in the feedforward network. num_heads (int): Number of attention heads. dropout (float): Dropout probability. act (nn.Module): Activation function. normalize_before (bool): Whether to apply normalization before attention and feedforward. N)rr)r,r-r.r/rr*r+r0r2r3r–s z AIFI.__init__csp|jdd…\}}}| |||¡}tƒj| d¡ ddd¡|j|j|jdd}| ddd¡  d|||g¡  ¡S)zæ Forward pass for the AIFI transformer layer. Args: x (torch.Tensor): Input tensor with shape [B, C, H, W]. Returns: (torch.Tensor): Output tensor with shape [B, C, H, W]. rNér)ÚdeviceÚdtype)r7éÿÿÿÿ) ÚshapeÚ"build_2d_sincos_position_embeddingrrFÚflattenÚpermuteÚtorQrRÚviewÚ contiguous)r,ÚxÚcÚhÚwÚ pos_embedr0r2r3rF¤s . z AIFI.forwardéçˆÃ@c CsÈ|ddks Jdƒ‚tj|tjd}tj|tjd}tj||dd\}}|d}tj|tjd|}d||}| ¡d|d }| ¡d|d } t t |¡t |¡t | ¡t | ¡gd ¡d S) a‡ Build 2D sine-cosine position embedding. Args: w (int): Width of the feature map. h (int): Height of the feature map. embed_dim (int): Embedding dimension. temperature (float): Temperature for the sine/cosine functions. Returns: (torch.Tensor): Position embedding with shape [1, embed_dim, h*w]. érzHEmbed dimension must be divisible by 4 for 2D sin-cos position embedding©rRÚij)Úindexinggð?©.NNr)ÚtorchÚarangeÚfloat32ÚmeshgridrVÚcatÚsinÚcos) r^r]Ú embed_dimÚ temperatureÚgrid_wÚgrid_hÚpos_dimÚomegaÚout_wÚout_hr2r2r3rU´s 0z'AIFI.build_2d_sincos_position_embedding)r`ra) rIrJrKrLrrMrrFrNrUrOr2r2r0r3rs  rcó(eZdZdZ‡fdd„Zdd„Z‡ZS)r zeTransformer layer https://arxiv.org/abs/2010.11929 (LayerNorm layers removed for better performance).csxtƒ ¡tj||dd|_tj||dd|_tj||dd|_tj||d|_tj||dd|_ tj||dd|_ dS)zî Initialize a self-attention mechanism using linear transformations and multi-head attention. Args: c (int): Input and output channel dimension. num_heads (int): Number of attention heads. F)Úbias)rnr/N) rrrr!rArBÚvrr r"r#)r,r\r/r0r2r3rÓs zTransformerLayer.__init__cCs<| | |¡| |¡| |¡¡d|}| | |¡¡|S)zá Apply a transformer block to the input x and return the output. Args: x (torch.Tensor): Input tensor. Returns: (torch.Tensor): Output tensor after transformer layer. r)r rArBrxr#r"©r,r[r2r2r3rFãs( zTransformerLayer.forward©rIrJrKrLrrFrOr2r2r0r3r Ðs r crv)r aS Vision Transformer https://arxiv.org/abs/2010.11929. Attributes: conv (Conv, optional): Convolution layer if input and output channels differ. linear (nn.Linear): Learnable position embedding. tr (nn.Sequential): Sequential container of transformer layers. c2 (int): Output channel dimension. cs\tƒ ¡d|_|ˆkrt|ˆƒ|_t ˆˆ¡|_tj‡‡fdd„t|ƒDƒŽ|_ ˆ|_ dS)aU Initialize a Transformer module with position embedding and specified number of heads and layers. Args: c1 (int): Input channel dimension. c2 (int): Output channel dimension. num_heads (int): Number of attention heads. num_layers (int): Number of transformer layers. Nc3s|]}tˆˆƒVqdSrG)r )Ú.0Ú_©Úc2r/r2r3Ú s€z,TransformerBlock.__init__..) rrÚconvrrr!ÚlinearÚ SequentialÚrangeÚtrr~)r,r-r~r/Ú num_layersr0r}r3rüs    zTransformerBlock.__init__cCsb|jdur | |¡}|j\}}}}| d¡ ddd¡}| || |¡¡ ddd¡ ||j||¡S)z÷ Forward propagates the input through the bottleneck module. Args: x (torch.Tensor): Input tensor with shape [b, c1, w, h]. Returns: (torch.Tensor): Output tensor with shape [b, c2, w, h]. NrPrr)r€rTrVrWr„rÚreshaper~)r,r[Úbr|r^r]Úpr2r2r3rFs ,zTransformerBlock.forwardrzr2r2r0r3r ñs r cs<eZdZdZejf‡fdd„ Zdejdejfdd„Z ‡Z S)r z6Implements a single block of a multi-layer perceptron.cs2tƒ ¡t ||¡|_t ||¡|_|ƒ|_dS)a  Initialize the MLPBlock with specified embedding dimension, MLP dimension, and activation function. Args: embedding_dim (int): Input and output dimension. mlp_dim (int): Hidden dimension. act (nn.Module): Activation function. N)rrrr!Úlin1Úlin2r*)r,Ú embedding_dimÚmlp_dimr*r0r2r3r"s  zMLPBlock.__init__r[ÚreturncCs| | | |¡¡¡S)z¸ Forward pass for the MLPBlock. Args: x (torch.Tensor): Input tensor. Returns: (torch.Tensor): Output tensor after MLP block. )rŠr*r‰ryr2r2r3rF0s zMLPBlock.forward) rIrJrKrLrrMrrgÚTensorrFrOr2r2r0r3r sr cs0eZdZdZejdf‡fdd„ Zdd„Z‡ZS)ra4 Implements a simple multi-layer perceptron (also called FFN). Attributes: num_layers (int): Number of layers in the MLP. layers (nn.ModuleList): List of linear layers. sigmoid (bool): Whether to apply sigmoid to the output. act (nn.Module): Activation function. FcsXtƒ ¡||_|g|d}t dd„t|g|||gƒDƒ¡|_||_|ƒ|_dS)a­ Initialize the MLP with specified input, hidden, output dimensions and number of layers. Args: input_dim (int): Input dimension. hidden_dim (int): Hidden dimension. output_dim (int): Output dimension. num_layers (int): Number of layers. act (nn.Module): Activation function. sigmoid (bool): Whether to apply sigmoid to the output. rcss |] \}}t ||¡VqdSrG)rr!)r{ÚnrBr2r2r3rWó€zMLP.__init__..N) rrr…rÚ ModuleListÚzipÚlayersÚsigmoidr*)r,Ú input_dimÚ hidden_dimÚ output_dimr…r*r”r]r0r2r3rHs ( z MLP.__init__cCsZt|jƒD]\}}||jdkrt|dt ¡ƒ||ƒƒn||ƒ}qt|ddƒr+| ¡S|S)z´ Forward pass for the entire MLP. Args: x (torch.Tensor): Input tensor. Returns: (torch.Tensor): Output tensor after MLP. rr*r”F)Ú enumerater“r…ÚgetattrrÚReLUr”)r,r[ÚiÚlayerr2r2r3rF[s 0z MLP.forward) rIrJrKrLrršrrFrOr2r2r0r3r=s rcs*eZdZdZd‡fdd„ Zdd„Z‡ZS)r að 2D Layer Normalization module inspired by Detectron2 and ConvNeXt implementations. Original implementations in https://github.com/facebookresearch/detectron2/blob/main/detectron2/layers/batch_norm.py and https://github.com/facebookresearch/ConvNeXt/blob/main/models/convnext.py. Attributes: weight (nn.Parameter): Learnable scale parameter. bias (nn.Parameter): Learnable bias parameter. eps (float): Small constant for numerical stability. çíµ ÷Æ°>cs8tƒ ¡t t |¡¡|_t t |¡¡|_||_ dS)zÔ Initialize LayerNorm2d with the given parameters. Args: num_channels (int): Number of channels in the input. eps (float): Small constant for numerical stability. N) rrrÚ ParameterrgÚonesÚweightÚzerosrwÚeps)r,Ú num_channelsr¢r0r2r3rys  zLayerNorm2d.__init__cCsj|jddd}|| d¡jddd}||t ||j¡}|jdd…ddf||jdd…ddfS)zÅ Perform forward pass for 2D layer normalization. Args: x (torch.Tensor): Input tensor. Returns: (torch.Tensor): Normalized output tensor. rT©ÚkeepdimrPN)ÚmeanÚpowrgÚsqrtr¢r rw)r,r[ÚuÚsr2r2r3rF†s ,zLayerNorm2d.forward)rrzr2r2r0r3r js r cs4eZdZdZd ‡fdd„ Zdd„Zd d d „Z‡ZS)ra; Multiscale Deformable Attention Module based on Deformable-DETR and PaddleDetection implementations. https://github.com/fundamentalvision/Deformable-DETR/blob/main/models/ops/modules/ms_deform_attn.py Attributes: im2col_step (int): Step size for im2col operations. d_model (int): Model dimension. n_levels (int): Number of feature levels. n_heads (int): Number of attention heads. n_points (int): Number of sampling points per attention head per feature level. sampling_offsets (nn.Linear): Linear layer for generating sampling offsets. attention_weights (nn.Linear): Linear layer for generating attention weights. value_proj (nn.Linear): Linear layer for projecting values. output_proj (nn.Linear): Linear layer for projecting output. r`rbrcs¼tƒ ¡||dkrtd|›d|›ƒ‚||}|||ks#Jdƒ‚d|_||_||_||_||_t  ||||d¡|_ t  ||||¡|_ t  ||¡|_ t  ||¡|_ | ¡dS)aG Initialize MSDeformAttn with the given parameters. Args: d_model (int): Model dimension. n_levels (int): Number of feature levels. n_heads (int): Number of attention heads. n_points (int): Number of sampling points per attention head per feature level. rz.d_model must be divisible by n_heads, but got z and z(`d_model` must be divisible by `n_heads`é@rPN)rrÚ ValueErrorÚ im2col_stepÚd_modelÚn_levelsÚn_headsÚn_pointsrr!Úsampling_offsetsÚattention_weightsÚ value_projÚ output_projÚ_reset_parameters)r,r®r¯r°r±Ú _d_per_headr0r2r3r¨s  zMSDeformAttn.__init__cCsVt|jjjdƒtj|jtjddtj |j}t  |  ¡|  ¡gd¡}||  ¡jdddd |jddd ¡ d|j|jd¡}t|jƒD]}|d d …d d …|d d …f|d9<qFt ¡t | d¡¡|j_Wd ƒn1svwYt|jjjdƒt|jjjdƒt|jjjƒt|jjjdƒt|jjjƒt|jjjdƒd S) zReset module parameters.rrcg@rSTr¤rrrPN)rr²r Údatargrhr°riÚmathÚpiÚstackrmrlÚabsÚmaxrYÚrepeatr¯r±rƒÚno_gradrržrwr³rr´rµ)r,ÚthetasÚ grid_initr›r2r2r3r¶Çs$"ý* ÿzMSDeformAttn._reset_parametersNc CsØ|jdd…\}}|jd}tdd„|Dƒƒ|ksJ‚| |¡}|dur.| |dtdƒ¡}| |||j|j|j¡}| |¡ |||j|j |j d¡} |  |¡ |||j|j |j ¡} t   | d¡ |||j|j |j ¡} |jd} | dkr¦tj||j|jd  d¡} | | ddddd…ddd…f} |dd…dd…ddd…ddd…f| }n:| d krØ| |j |dd…dd…ddd…ddd…fd } |dd…dd…ddd…ddd…f| }ntd | ›d ƒ‚t|||| ƒ}| |¡S)a™ Perform forward pass for multiscale deformable attention. https://github.com/PaddlePaddle/PaddleDetection/blob/develop/ppdet/modeling/transformers/deformable_transformer.py Args: query (torch.Tensor): Tensor with shape [bs, query_length, C]. refer_bbox (torch.Tensor): Tensor with shape [bs, query_length, n_levels, 2], range in [0, 1], top-left (0,0), bottom-right (1, 1), including padding area. value (torch.Tensor): Tensor with shape [bs, value_length, C]. value_shapes (list): List with shape [n_levels, 2], [(H_0, W_0), (H_1, W_1), ..., (H_{L-1}, W_{L-1})]. value_mask (torch.Tensor, optional): Tensor with shape [bs, value_length], True for non-padding elements, False for padding elements. Returns: (torch.Tensor): Output tensor with shape [bs, Length_{query}, C]. NrPrcss |] }|d|dVqdS)rrNr2)r{rªr2r2r3rðrz'MSDeformAttn.forward..rfrrS)rRrQrbgà?z5Last dim of reference_points must be 2 or 4, but got Ú.)rTÚsumr´Ú masked_fillÚfloatrYr°r®r²r¯r±r³ÚFÚsoftmaxrgÚ as_tensorrRrQÚflipr¬rrµ)r,ÚqueryÚ refer_bboxr;Ú value_shapesÚ value_maskÚbsÚlen_qÚlen_vr²r³Ú num_pointsÚoffset_normalizerÚaddÚsampling_locationsÚoutputr2r2r3rFÜs*       *2* zMSDeformAttn.forward)r`rbrrbrG)rIrJrKrLrr¶rFrOr2r2r0r3r–s rcsReZdZdZdddde ¡ddf‡fdd„ Zed d „ƒZd d „Z ddd„Z ‡Z S)ra† Deformable Transformer Decoder Layer inspired by PaddleDetection and Deformable-DETR implementations. https://github.com/PaddlePaddle/PaddleDetection/blob/develop/ppdet/modeling/transformers/deformable_transformer.py https://github.com/fundamentalvision/Deformable-DETR/blob/main/models/deformable_transformer.py Attributes: self_attn (nn.MultiheadAttention): Self-attention module. dropout1 (nn.Dropout): Dropout after self-attention. norm1 (nn.LayerNorm): Layer normalization after self-attention. cross_attn (MSDeformAttn): Cross-attention module. dropout2 (nn.Dropout): Dropout after cross-attention. norm2 (nn.LayerNorm): Layer normalization after cross-attention. linear1 (nn.Linear): First linear layer in the feedforward network. act (nn.Module): Activation function. dropout3 (nn.Dropout): Dropout in the feedforward network. linear2 (nn.Linear): Second linear layer in the feedforward network. dropout4 (nn.Dropout): Dropout after the feedforward network. norm3 (nn.LayerNorm): Layer normalization after the feedforward network. r`rirrbcs¦tƒ ¡tj|||d|_t |¡|_t |¡|_t ||||ƒ|_ t |¡|_ t |¡|_ t  ||¡|_||_t |¡|_t  ||¡|_t |¡|_t |¡|_dS)aÞ Initialize the DeformableTransformerDecoderLayer with the given parameters. Args: d_model (int): Model dimension. n_heads (int): Number of attention heads. d_ffn (int): Dimension of the feedforward network. dropout (float): Dropout probability. act (nn.Module): Activation function. n_levels (int): Number of feature levels. n_points (int): Number of sampling points. )rN)rrrrÚ self_attnr'r(r$r%rÚ cross_attnr)r&r!Úlinear1r*Údropout3Úlinear2Údropout4Únorm3)r,r®r°Úd_ffnrr*r¯r±r0r2r3rs       z*DeformableTransformerDecoderLayer.__init__cCr4)z;Add positional embeddings to the input tensor, if provided.Nr2r5r2r2r3r8?r9z0DeformableTransformerDecoderLayer.with_pos_embedc Cs4| | | | |¡¡¡¡}|| |¡}| |¡S)zÞ Perform forward pass through the Feed-Forward Network part of the layer. Args: tgt (torch.Tensor): Input tensor. Returns: (torch.Tensor): Output tensor after FFN. )rÚrÙr*rØrÛrÜ)r,ÚtgtÚtgt2r2r2r3Ú forward_ffnDs  z-DeformableTransformerDecoderLayer.forward_ffnNc Cs | ||¡}} |j| dd¡|  dd¡| dd¡|dd dd¡} || | ¡}| |¡}| | ||¡| d¡|||¡} || | ¡}| |¡}|  |¡S)aH Perform the forward pass through the entire decoder layer. Args: embed (torch.Tensor): Input embeddings. refer_bbox (torch.Tensor): Reference bounding boxes. feats (torch.Tensor): Feature maps. shapes (list): Feature shapes. padding_mask (torch.Tensor, optional): Padding mask. attn_mask (torch.Tensor, optional): Attention mask. query_pos (torch.Tensor, optional): Query position embeddings. Returns: (torch.Tensor): Output tensor after decoder layer. rr)r<rP) r8rÖÚ transposer(r%r×Ú unsqueezer)r&rà) r,ÚembedrËÚfeatsÚshapesÚ padding_maskr<Ú query_posrArBrÞr2r2r3rFRs(ÿþ ÿ  z)DeformableTransformerDecoderLayer.forwardrH) rIrJrKrLrršrrNr8ràrFrOr2r2r0r3rs ! rcs0eZdZdZd‡fdd„ Z  d dd„Z‡ZS) raº Implementation of Deformable Transformer Decoder based on PaddleDetection. https://github.com/PaddlePaddle/PaddleDetection/blob/develop/ppdet/modeling/transformers/deformable_transformer.py Attributes: layers (nn.ModuleList): List of decoder layers. num_layers (int): Number of decoder layers. hidden_dim (int): Hidden dimension. eval_idx (int): Index of the layer to use during evaluation. rScsBtƒ ¡t||ƒ|_||_||_|dkr||_dS|||_dS)aU Initialize the DeformableTransformerDecoder with the given parameters. Args: hidden_dim (int): Hidden dimension. decoder_layer (nn.Module): Decoder layer module. num_layers (int): Number of decoder layers. eval_idx (int): Index of the layer to use during evaluation. rN)rrrr“r…r–Úeval_idx)r,r–Ú decoder_layerr…rèr0r2r3r‚s  z%DeformableTransformerDecoder.__init__Nc  Csü|} g} g} d} | ¡}t|jƒD]b\}}|| |||| |||ƒƒ} ||| ƒ}t |t|ƒ¡}|jrS|  ||| ƒ¡|dkrF|  |¡n"|  t |t| ƒ¡¡n||jkrh|  ||| ƒ¡|  |¡n |} |jrq| ¡n|}qt  | ¡t  | ¡fS)aé Perform the forward pass through the entire decoder. Args: embed (torch.Tensor): Decoder embeddings. refer_bbox (torch.Tensor): Reference bounding boxes. feats (torch.Tensor): Image features. shapes (list): Feature shapes. bbox_head (nn.Module): Bounding box prediction head. score_head (nn.Module): Score prediction head. pos_mlp (nn.Module): Position MLP. attn_mask (torch.Tensor, optional): Attention mask. padding_mask (torch.Tensor, optional): Padding mask. Returns: dec_bboxes (torch.Tensor): Decoded bounding boxes. dec_cls (torch.Tensor): Decoded classification scores. Nr) r”r˜r“rgrÚtrainingÚappendrèÚdetachr»)r,rãrËräråÚ bbox_headÚ score_headÚpos_mlpr<rærÕÚ dec_bboxesÚdec_clsÚlast_refined_bboxr›rœÚbboxÚ refined_bboxr2r2r3rF’s*    z$DeformableTransformerDecoder.forward)rS)NNrzr2r2r0r3rus  ör)rLr¹rgÚtorch.nnrÚtorch.nn.functionalÚ functionalrÆÚ torch.nn.initrrr€rÚutilsrrrÚ__all__ÚModuler rr r r rr rrrr2r2r2r3Ús&  sA!.-,rm