o V h`}@sddlZddlmZddlmZddlmZmZmZm Z m Z m Z ddl Z ddlZddlmmZddlmZmZddlmZmZmZddlmZddlmZmZdd lmZmZdd l m!Z!dd l"m#Z#m$Z$dd l%m&Z&gd Z'de e(e(fde(de(de(de e(e(ff ddZ)de e(e(fde(dee e(e(ffddZ*de(de(dejfddZ+Gdddej,Z-Gdddej,Z.Gd d!d!ej,Z/Gd"d#d#ej,Z0Gd$d%d%ej,Z1Gd&d'd'ej,Z2Gd(d)d)ej,Z3Gd*d+d+ej,Z4Gd,d-d-ej,Z5  .dBd/e(d0ee(d1ee(d2e6d3e(d4e(d5e ed6e7d7ede5fd8d9Z8Gd:d;d;eZ9eedd?d5e e9d6e7d7ede5fd@dAZ;dS)CN) OrderedDict)partial)AnyCallableListOptionalSequenceTuple)nnTensor)register_modelWeights WeightsEnum)_IMAGENET_CATEGORIES)_ovewrite_named_paramhandle_legacy_interface)Conv2dNormActivationSqueezeExcitation)StochasticDepth)ImageClassificationInterpolationMode)_log_api_usage_once)MaxVitMaxVit_T_Weightsmaxvit_t input_size kernel_sizestridepaddingreturncCs8|d|d||d|d|d||dfS)Nr)rrrrr"r"M/var/www/vscode/kcb/lib/python3.10/site-packages/torchvision/models/maxvit.py_get_conv_output_shapesr$n_blockscCs<g}t|ddd}t|D]}t|ddd}||q |S)zQUtil function to check that the input size is correct for a MaxVit configuration.r r!)r$rangeappend)rr%shapesblock_input_shape_r"r"r#_make_block_input_shapes s   r,heightwidthcCsttjt|t|gdd}t|d}|dddddf|dddddf}|ddd}|dddddf|d7<|dddddf|d7<|dddddfd|d9<|dS)Nij)indexingr!r r)torchstackmeshgridarangeflattenpermute contiguoussum)r-r.coords coords_flatrelative_coordsr"r"r#_get_relative_position_index*s$ ,""& r=cspeZdZdZ ddedededededed ejfd ed ejfd ed d ffdd Z de d e fddZ Z S)MBConva=MBConv: Mobile Inverted Residual Bottleneck. Args: in_channels (int): Number of input channels. out_channels (int): Number of output channels. expansion_ratio (float): Expansion ratio in the bottleneck. squeeze_ratio (float): Squeeze ratio in the SE Layer. stride (int): Stride of the depthwise convolution. activation_layer (Callable[..., nn.Module]): Activation function. norm_layer (Callable[..., nn.Module]): Normalization function. p_stochastic_dropout (float): Probability of stochastic depth. in_channels out_channelsexpansion_ratio squeeze_ratioractivation_layer. norm_layerp_stochastic_dropoutrNc  s*t||dkp||k} | r2tj||ddddg} |dkr+tjd|ddg| } tj| |_nt|_t||} t||} |rMt |dd|_ nt|_ t } ||| d <t || ddd ||dd | d <t | | d|d||| dd | d<t | | tjd| d<tj| |ddd| d<t| |_dS)Nr!T)rrbiasr r&rrrrowmodepre_normr)rrrrDrEinplaceconv_a)rrrrDrEgroupsrMconv_b) activationsqueeze_excitation)r@rArrGconv_c)super__init__r Conv2d AvgPool2d SequentialprojIdentityintrstochastic_depthrrrSiLUlayers)selfr@rArBrCrrDrErF should_projrY mid_channels sqz_channels_layers __class__r"r#rUCsP          zMBConv.__init__xcCs"||}|||}||S)z Args: x (Tensor): Input tensor with expected layout of [B, C, H, W]. Returns: Tensor: Output tensor with expected layout of [B, C, H / stride, W / stride]. )rYr\r^r_rfresr"r"r#forwards zMBConv.forward)r?) __name__ __module__ __qualname____doc__r[floatrr ModulerUr ri __classcell__r"r"rdr#r>5s.    =r>csTeZdZdZdedededdffdd Zdejfd d Zd edefd d Z Z S)$RelativePositionalMultiHeadAttentionzRelative Positional Multi-Head Attention. Args: feat_dim (int): Number of input features. head_dim (int): Number of features per head. max_seq_len (int): Maximum sequence length. feat_dimhead_dim max_seq_lenrNcst||dkrtd|d||||_||_tt||_||_ t ||j|jd|_ |d|_ t |j|j||_t jtjd|jdd|jd|jftjd|_|d t|j|jtj jj|jd d dS) Nrz feat_dim: z must be divisible by head_dim: r&gr r!)dtyperelative_position_index{Gz?std)rTrU ValueErrorn_headsrsr[mathsqrtsizertr Linearto_qkv scale_factormerge parameter Parameterr2emptyfloat32relative_position_bias_tableregister_bufferr=init trunc_normal_)r_rrrsrtrdr"r#rUs    ,z-RelativePositionalMultiHeadAttention.__init__cCs@|jd}|j||j|jd}|ddd}|dS)Nr1r rr!)rvviewrrtr7r8 unsqueeze)r_ bias_index relative_biasr"r"r#get_relative_positional_biass  zARelativePositionalMultiHeadAttention.get_relative_positional_biasrfcCs|j\}}}}|j|j}}||}tj|ddd\} } } | |||||ddddd} | |||||ddddd} | |||||ddddd} | |j} t d| | } | } t j | | dd} t d | | }|ddddd||||}| |}|S) z Args: x (Tensor): Input tensor with expected layout of [B, G, P, D]. Returns: Tensor: Output tensor with expected layout of [B, G, P, D]. r&r1)dimrr!r z!B G H I D, B G H J D -> B G H I Jz!B G H I J, B G H J D -> B G H I D)shaper{rsrr2chunkreshaper7reinsumrFsoftmaxr)r_rfBGPDHDHqkvqkvdot_prodpos_biasoutr"r"r#ris      z,RelativePositionalMultiHeadAttention.forward) rjrkrlrmr[rUr2r rrirpr"r"rdr#rqsrqcsDeZdZdZdededdffdd Zdejdejfd d ZZ S) SwapAxeszPermute the axes of a tensor.abrNcst||_||_dSN)rTrUrr)r_rrrdr"r#rUs  zSwapAxes.__init__rfcCst||j|j}|Sr)r2swapaxesrrrgr"r"r#riszSwapAxes.forward) rjrkrlrmr[rUr2r rirpr"r"rdr#rsrcs8eZdZdZd fdd Zdededefdd ZZS) WindowPartitionzB Partition the input tensor into non-overlapping windows. rNctdSrrTrUr_rdr"r#rUzWindowPartition.__init__rfpcCsf|j\}}}}|}|||||||||}|dddddd}|||||||||}|S)z Args: x (Tensor): Input tensor with expected layout of [B, C, H, W]. p (int): Number of partitions. Returns: Tensor: Output tensor with expected layout of [B, H/P, W/P, P*P, C]. rr rr&r!rrr7)r_rfrrCrWrr"r"r#ris  zWindowPartition.forwardrN rjrkrlrmrUr r[rirpr"r"rdr#rsrc s@eZdZdZd fdd Zdededed edef d d ZZS) WindowDepartitionzo Departition the input tensor of non-overlapping windows into a feature volume of layout [B, C, H, W]. rNcrrrrrdr"r#rUrzWindowDepartition.__init__rfr h_partitions w_partitionsc Cs`|j\}}}}|} ||} } ||| | | | |}|dddddd}|||| | | | }|S)ar Args: x (Tensor): Input tensor with expected layout of [B, (H/P * W/P), P*P, C]. p (int): Number of partitions. h_partitions (int): Number of vertical partitions. w_partitions (int): Number of horizontal partitions. Returns: Tensor: Output tensor with expected layout of [B, C, H, W]. rrr!r&r rr) r_rfrrrrrPPrrHPWPr"r"r#ris  zWindowDepartition.forwardrrr"r"rdr#rs&rcseZdZdZdededededeeefdeded ej fd ed ej fd e d e d e ddffdd Z de de fddZ ZS)PartitionAttentionLayera Layer for partitioning the input tensor into non-overlapping windows and applying attention to each window. Args: in_channels (int): Number of input channels. head_dim (int): Dimension of each attention head. partition_size (int): Size of the partitions. partition_type (str): Type of partitioning to use. Can be either "grid" or "window". grid_size (Tuple[int, int]): Size of the grid to partition the input tensor into. mlp_ratio (int): Ratio of the feature size expansion in the MLP layer. activation_layer (Callable[..., nn.Module]): Activation function to use. norm_layer (Callable[..., nn.Module]): Normalization function to use. attention_dropout (float): Dropout probability for the attention layer. mlp_dropout (float): Dropout probability for the MLP layer. p_stochastic_dropout (float): Probability of dropping out a partition. r@rspartition_sizepartition_type grid_size mlp_ratiorD.rEattention_dropout mlp_dropoutrFrNc s(t|||_||_|d||_||_||_|dvr"td|dkr/||j|_|_ n|j||_|_ t |_ t |_ |dkrHtddnt|_|dkrVtddnt|_t||t|||dt| |_tt|t||||t|||t| |_t| d d |_dS) Nr)gridwindowz0partition_type must be either 'grid' or 'window'rrr rIrJ)rTrUr{rs n_partitionsrrrzrgr partition_oprdepartition_oprr rZpartition_swapdepartition_swaprXrqDropout attn_layer LayerNormr mlp_layerrstochastic_dropout) r_r@rsrrrrrDrErrrFrdr"r#rU,s8    z PartitionAttentionLayer.__init__rfcCs|jd|j|jd|j}}t|jd|jdko&|jd|jdkd|j|j|||j}||}||||}||| |}| |}| ||j||}|S)z Args: x (Tensor): Input tensor with expected layout of [B, C, H, W]. Returns: Tensor: Output tensor with expected layout of [B, C, H, W]. rr!z[Grid size must be divisible by partition size. Got grid size of {} and partition size of {}) rrr2_assertformatrrrrrrr)r_rfghgwr"r"r#rifs" &  zPartitionAttentionLayer.forward)rjrkrlrmr[strr rr rornrUr rirpr"r"rdr#rs8     :rcseZdZdZdededededededejfd edejfd ed ed ed ededede eefddffdd Z de de fddZ Z S) MaxVitLayera MaxVit layer consisting of a MBConv layer followed by a PartitionAttentionLayer with `window` and a PartitionAttentionLayer with `grid`. Args: in_channels (int): Number of input channels. out_channels (int): Number of output channels. expansion_ratio (float): Expansion ratio in the bottleneck. squeeze_ratio (float): Squeeze ratio in the SE Layer. stride (int): Stride of the depthwise convolution. activation_layer (Callable[..., nn.Module]): Activation function. norm_layer (Callable[..., nn.Module]): Normalization function. head_dim (int): Dimension of the attention heads. mlp_ratio (int): Ratio of the MLP layer. mlp_dropout (float): Dropout probability for the MLP layer. attention_dropout (float): Dropout probability for the attention layer. p_stochastic_dropout (float): Probability of stochastic depth. partition_size (int): Size of the partitions. grid_size (Tuple[int, int]): Size of the input feature grid. r@rArCrBrrE.rDrsrrrrFrrrNc stt}t|||||||| d|d<t||| d|| |tj| | | d |d<t||| d|| |tj| | | d |d<t||_dS)N)r@rArBrCrrDrErFMBconvr) r@rsrrrrrDrErrrFwindow_attentionrgrid_attention) rTrUrr>rr rrXr^)r_r@rArCrBrrErDrsrrrrFrrr^rdr"r#rUsN       zMaxVitLayer.__init__rfcCs||}|Sz Args: x (Tensor): Input tensor of shape (B, C, H, W). Returns: Tensor: Output tensor of shape (B, C, H, W). r^)r_rfr"r"r#ris zMaxVitLayer.forward)rjrkrlrmr[rnrr ror rUr rirpr"r"rdr#rsD     ArcseZdZdZdedededededejfdedejfd ed ed ed ed ede eefdede eddffdd Z de de fddZ ZS) MaxVitBlocka( A MaxVit block consisting of `n_layers` MaxVit layers. Args: in_channels (int): Number of input channels. out_channels (int): Number of output channels. expansion_ratio (float): Expansion ratio in the bottleneck. squeeze_ratio (float): Squeeze ratio in the SE Layer. activation_layer (Callable[..., nn.Module]): Activation function. norm_layer (Callable[..., nn.Module]): Normalization function. head_dim (int): Dimension of the attention heads. mlp_ratio (int): Ratio of the MLP layer. mlp_dropout (float): Dropout probability for the MLP layer. attention_dropout (float): Dropout probability for the attention layer. p_stochastic_dropout (float): Probability of stochastic depth. partition_size (int): Size of the partitions. input_grid_size (Tuple[int, int]): Size of the input feature grid. n_layers (int): Number of layers in the block. p_stochastic (List[float]): List of probabilities for stochastic depth for each layer. r@rArCrBrE.rDrsrrrrinput_grid_sizen_layers p_stochasticrNcstt|| kstd| d|dt|_t| dddd|_t |D]+\}}|dkr2dnd}|jt |dkr>|n|||||||||| | | |j|d g7_q(dS) Nz'p_stochastic must have length n_layers=z, got p_stochastic=.r&r r!rHr)r@rArCrBrrErDrsrrrrrrF) rTrUlenrzr ModuleListr^r$r enumerater)r_r@rArCrBrErDrsrrrrrrridxrrrdr"r#rUs4    zMaxVitBlock.__init__rfcCs|jD]}||}q|Srr)r_rflayerr"r"r#ri,s  zMaxVitBlock.forward)rjrkrlrmr[rnrr ror rrUr rirpr"r"rdr#rsD      3rc!seZdZdZdejddddddfdeeefded ed eed eed ed e de e dej fde dej fde de dede de deddf fdd Z dedefddZddZZS)ray Implements MaxVit Transformer from the `MaxViT: Multi-Axis Vision Transformer `_ paper. Args: input_size (Tuple[int, int]): Size of the input image. stem_channels (int): Number of channels in the stem. partition_size (int): Size of the partitions. block_channels (List[int]): Number of channels in each block. block_layers (List[int]): Number of layers in each block. stochastic_depth_prob (float): Probability of stochastic depth. Expands to a list of probabilities for each layer that scales linearly to the specified value. squeeze_ratio (float): Squeeze ratio in the SE Layer. Default: 0.25. expansion_ratio (float): Expansion ratio in the MBConv bottleneck. Default: 4. norm_layer (Callable[..., nn.Module]): Normalization function. Default: None (setting to None will produce a `BatchNorm2d(eps=1e-3, momentum=0.01)`). activation_layer (Callable[..., nn.Module]): Activation function Default: nn.GELU. head_dim (int): Dimension of the attention heads. mlp_ratio (int): Expansion ratio of the MLP layer. Default: 4. mlp_dropout (float): Dropout probability for the MLP layer. Default: 0.0. attention_dropout (float): Dropout probability for the attention layer. Default: 0.0. num_classes (int): Number of classes. Default: 1000. Ng?rr?ir stem_channelsrblock_channels block_layersrsstochastic_depth_probrE.rDrCrBrrr num_classesrcstt|d}|durttjddd}t|t|}t|D]%\}}|d|dks6|d|dkrGt d|d|d |d |d q"t t ||dd || d ddt ||dddddd|_ t |dd dd}||_t|_|g|dd}|}td|t|}d}t|||D]+\}}}|jt||| | || || | ||||||||d|jdj}||7}qt tdtt|dt|d|dttj|d|d d|_|dS)Nr&gMbP?g{Gz?)epsmomentumrr!z Input size z of block z$ is not divisible by partition size zx. Consider changing the partition size or the input size. Current configuration yields the following block input sizes: rr F)rrErDrGrMT)rrErDrGrHr1)r@rArCrBrErDrsrrrrrrr)rG) rTrUrrr BatchNorm2dr,rrrzrXrstemr$rrblocksnplinspacer9tolistzipr(rrAdaptiveAvgPool2dFlattenrrTanh classifier _init_weights)r_rrrrrrsrrErDrCrBrrrrinput_channelsblock_input_sizesrblock_input_sizer@rArp_idx in_channel out_channel num_layersrdr"r#rUMs        zMaxVit.__init__rfcCs,||}|jD]}||}q||}|Sr)rrr)r_rfblockr"r"r#ris    zMaxVit.forwardcCs|D]P}t|tjr"tjj|jdd|jdur!tj|jqt|tj r9tj |jdtj |jdqt|tj rTtjj|jdd|jdurTtj|jqdS)Nrwrxr!r) modules isinstancer rVrnormal_weightrGzeros_r constant_r)r_mr"r"r#rs      zMaxVit._init_weights)rjrkrlrmr GELUr r[rrnrrrorUr rirrpr"r"rdr#r8sZ%      vrFrrrrrrsweightsprogresskwargsc Ks|dur(t|dt|jd|jdd|jddksJt|d|jd|dd} td ||||||| d|} |durK| |j|d d | S) Nr categoriesmin_sizerr!rr)rrrrrsrrT)r  check_hashr")rrmetapoprload_state_dictget_state_dict) rrrrrrsr r r rmodelr"r"r#_maxvits&   rc @sHeZdZedeeddejdedddddd d id d d ddZ e Z dS)rz9https://download.pytorch.org/models/maxvit_t-bc5ab103.pthr) crop_size resize_size interpolationirzLhttps://github.com/pytorch/vision/tree/main/references/classification#maxvitz ImageNet-1KgT@g|?5.X@)zacc@1zacc@5gZd;@gK7]@zThese weights reproduce closely the results of the paper using a similar training recipe. They were trained with a BatchNorm2D momentum of 0.99 instead of the more correct 0.01.)r  num_paramsrrecipe_metrics_ops _file_size_docs)url transformsrN) rjrkrlr rrrBICUBICr IMAGENET1K_V1DEFAULTr"r"r"r#rs* r pretrained)r T)r r c Ks2t|}td dgdgdddd||d|S) a Constructs a maxvit_t architecture from `MaxViT: Multi-Axis Vision Transformer `_. Args: weights (:class:`~torchvision.models.MaxVit_T_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.MaxVit_T_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.maxvit.MaxVit`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.MaxVit_T_Weights :members: @)r'i)r r rr g?)rrrrsrrr r Nr")rverifyr)r r r r"r"r#rs  r)NF)rqrrrrrrrrnboolrrr$rr"r"r"r#sp      .*  WI haU.    *.