o W h@s ddlZddlmZddlZddlmZddlmmZddl m m Z ddl m Z ddlmZGdddejjZGdddejZGd d d ejZGd d d ejZGd ddejZGdddejZGdddejjZGdddejZGdddejZGdddejZdS)N)Tuple) LayerNorm2d) to_2tuplecs"eZdZdZdfdd ZZS) Conv2d_BNa A sequential container that performs 2D convolution followed by batch normalization. Attributes: c (torch.nn.Conv2d): 2D convolution layer. bn (torch.nn.BatchNorm2d): Batch normalization layer. Methods: __init__: Initializes the Conv2d_BN with specified parameters. Args: a (int): Number of input channels. b (int): Number of output channels. ks (int): Kernel size for the convolution. Defaults to 1. stride (int): Stride for the convolution. Defaults to 1. pad (int): Padding for the convolution. Defaults to 0. dilation (int): Dilation factor for the convolution. Defaults to 1. groups (int): Number of groups for the convolution. Defaults to 1. bn_weight_init (float): Initial value for batch normalization weight. Defaults to 1. Examples: >>> conv_bn = Conv2d_BN(3, 64, ks=3, stride=1, pad=1) >>> input_tensor = torch.randn(1, 3, 224, 224) >>> output = conv_bn(input_tensor) >>> print(output.shape) rc snt|dtjj|||||||ddtj|} tjj| j |tjj| j d|d| dS)zWInitializes a sequential container with 2D convolution followed by batch normalization.cF)biasrbnN) super__init__ add_moduletorchnnConv2d BatchNorm2dinit constant_weightr) selfabksstridepaddilationgroupsbn_weight_initr  __class___/var/www/vscode/kcb/lib/python3.10/site-packages/ultralytics/models/sam/modules/tiny_encoder.pyr 4s $ zConv2d_BN.__init__)rrrrrr)__name__ __module__ __qualname____doc__r __classcell__rrrr rsrc(eZdZdZfddZddZZS) PatchEmbeda Embeds images into patches and projects them into a specified embedding dimension. Attributes: patches_resolution (Tuple[int, int]): Resolution of the patches after embedding. num_patches (int): Total number of patches. in_chans (int): Number of input channels. embed_dim (int): Dimension of the embedding. seq (nn.Sequential): Sequence of convolutional and activation layers for patch embedding. Methods: forward: Processes the input tensor through the patch embedding sequence. Examples: >>> import torch >>> patch_embed = PatchEmbed(in_chans=3, embed_dim=96, resolution=224, activation=nn.GELU) >>> x = torch.randn(1, 3, 224, 224) >>> output = patch_embed(x) >>> print(output.shape) c stt|}|dd|ddf|_|jd|jd|_||_||_|}tt ||dddd|t |d|ddd|_ dS)zcInitializes patch embedding with convolutional layers for image-to-patch conversion and projection.rrN) r r rpatches_resolution num_patchesin_chans embed_dimr Sequentialrseq)rr-r. resolution activationimg_sizenrrr r Ts  zPatchEmbed.__init__cC ||S)z_Processes input tensor through patch embedding sequence, converting images to patch embeddings.)r0rxrrr forwardc zPatchEmbed.forwardr!r"r#r$r r8r%rrrr r'>s r'cr&)MBConva  Mobile Inverted Bottleneck Conv (MBConv) layer, part of the EfficientNet architecture. Attributes: in_chans (int): Number of input channels. hidden_chans (int): Number of hidden channels. out_chans (int): Number of output channels. conv1 (Conv2d_BN): First convolutional layer. act1 (nn.Module): First activation function. conv2 (Conv2d_BN): Depthwise convolutional layer. act2 (nn.Module): Second activation function. conv3 (Conv2d_BN): Final convolutional layer. act3 (nn.Module): Third activation function. drop_path (nn.Module): Drop path layer (Identity for inference). Methods: forward: Performs the forward pass through the MBConv layer. Examples: >>> in_chans, out_chans = 32, 64 >>> mbconv = MBConv(in_chans, out_chans, expand_ratio=4, activation=nn.ReLU, drop_path=0.1) >>> x = torch.randn(1, in_chans, 56, 56) >>> output = mbconv(x) >>> print(output.shape) torch.Size([1, 64, 56, 56]) cst||_t|||_||_t||jdd|_||_t|j|jddd|jd|_ ||_ t|j|ddd|_ ||_ t |_dS)zcInitializes the MBConv layer with specified input/output channels, expansion ratio, and activation.r)rr*rrrr)rrN)r r r-int hidden_chans out_chansrconv1act1conv2act2conv3act3rIdentity drop_path)rr-r@ expand_ratior2rHrrr r s zMBConv.__init__cCsR|}||}||}||}||}||}||}||7}||S)zQImplements the forward pass of MBConv, applying convolutions and skip connection.)rArBrCrDrErHrF)rr7shortcutrrr r8s       zMBConv.forwardr:rrrr r;hs r;cr&) PatchMerginga Merges neighboring patches in the feature map and projects to a new dimension. This class implements a patch merging operation that combines spatial information and adjusts the feature dimension. It uses a series of convolutional layers with batch normalization to achieve this. Attributes: input_resolution (Tuple[int, int]): The input resolution (height, width) of the feature map. dim (int): The input dimension of the feature map. out_dim (int): The output dimension after merging and projection. act (nn.Module): The activation function used between convolutions. conv1 (Conv2d_BN): The first convolutional layer for dimension projection. conv2 (Conv2d_BN): The second convolutional layer for spatial merging. conv3 (Conv2d_BN): The third convolutional layer for final projection. Methods: forward: Applies the patch merging operation to the input tensor. Examples: >>> input_resolution = (56, 56) >>> patch_merging = PatchMerging(input_resolution, dim=64, out_dim=128, activation=nn.ReLU) >>> x = torch.randn(4, 64, 56, 56) >>> output = patch_merging(x) >>> print(output.shape) csrt||_||_||_||_t||ddd|_|dvr!dnd}t||d|d|d|_t||ddd|_ dS)zcInitializes the PatchMerging module for merging and projecting neighboring patches in feature maps.rr>@@r)r*)rN) r r input_resolutiondimout_dimactrrArCrE)rrOrPrQr2stride_crrr r s zPatchMerging.__init__cCs||jdkr|j\}}t|}||||ddddd}||}||}||}||}||}| d ddS)zHApplies patch merging and dimension projection to the input feature map.r*rrr)) ndimrOlenviewpermuterArRrCrEflatten transpose)rr7HWBrrr r8s       zPatchMerging.forwardr:rrrr rKs  rKcs4eZdZdZ     d fdd Zdd ZZS) ConvLayera Convolutional Layer featuring multiple MobileNetV3-style inverted bottleneck convolutions (MBConv). This layer optionally applies downsample operations to the output and supports gradient checkpointing. Attributes: dim (int): Dimensionality of the input and output. input_resolution (Tuple[int, int]): Resolution of the input image. depth (int): Number of MBConv layers in the block. use_checkpoint (bool): Whether to use gradient checkpointing to save memory. blocks (nn.ModuleList): List of MBConv layers. downsample (Optional[Callable]): Function for downsampling the output. Methods: forward: Processes the input through the convolutional layers. Examples: >>> input_tensor = torch.randn(1, 64, 56, 56) >>> conv_layer = ConvLayer(64, (56, 56), depth=3, activation=nn.ReLU) >>> output = conv_layer(input_tensor) >>> print(output.shape) r=NF@c snt|_||_||_||_tfddt|D|_ |dur,d|_ dS|||d|_ dS)a Initializes the ConvLayer with the given dimensions and settings. This layer consists of multiple MobileNetV3-style inverted bottleneck convolutions (MBConv) and optionally applies downsampling to the output. Args: dim (int): The dimensionality of the input and output. input_resolution (Tuple[int, int]): The resolution of the input image. depth (int): The number of MBConv layers in the block. activation (nn.Module): Activation function applied after each convolution. drop_path (float | List[float]): Drop path rate. Single float or a list of floats for each MBConv. downsample (Optional[nn.Module]): Function for downsampling the output. None to skip downsampling. use_checkpoint (bool): Whether to use gradient checkpointing to save memory. out_dim (Optional[int]): The dimensionality of the output. None means it will be the same as `dim`. conv_expand_ratio (float): Expansion ratio for the MBConv layers. Examples: >>> input_tensor = torch.randn(1, 64, 56, 56) >>> conv_layer = ConvLayer(64, (56, 56), depth=3, activation=nn.ReLU) >>> output = conv_layer(input_tensor) >>> print(output.shape) c s.g|]}tttr|nqSr)r; isinstancelist.0ir2conv_expand_ratiorPrHrr !sz&ConvLayer.__init__..NrPrQr2 r r rPrOdepthuse_checkpointr ModuleListrangeblocks downsample) rrPrOrjr2rHrorkrQrfrrer r s #zConvLayer.__init__cC>|jD]}|jrt||n||}q|jdur|S||S)z_Processes input through convolutional layers, applying MBConv blocks and optional downsampling.Nrnrk checkpointrorr7blkrrr r84 zConvLayer.forward)r=NFNr_r:rrrr r^s>r^cs4eZdZdZddejdffdd ZddZZS)Mlpa Multi-layer Perceptron (MLP) module for transformer architectures. This module applies layer normalization, two fully-connected layers with an activation function in between, and dropout. It is commonly used in transformer-based architectures. Attributes: norm (nn.LayerNorm): Layer normalization applied to the input. fc1 (nn.Linear): First fully-connected layer. fc2 (nn.Linear): Second fully-connected layer. act (nn.Module): Activation function applied after the first fully-connected layer. drop (nn.Dropout): Dropout layer applied after the activation function. Methods: forward: Applies the MLP operations on the input tensor. Examples: >>> import torch >>> from torch import nn >>> mlp = Mlp(in_features=256, hidden_features=512, out_features=256, act_layer=nn.GELU, drop=0.1) >>> x = torch.randn(32, 100, 256) >>> output = mlp(x) >>> print(output.shape) torch.Size([32, 100, 256]) Nr=csZt|p|}|p |}t||_t|||_t|||_||_t ||_ dS)z\Initializes a multi-layer perceptron with configurable input, hidden, and output dimensions.N) r r r LayerNormnormLinearfc1fc2rRDropoutdrop)r in_featureshidden_features out_features act_layerr}rrr r Vs  z Mlp.__init__cCs<||}||}||}||}||}||S)z[Applies MLP operations: layer norm, FC layers, activation, and dropout to the input tensor.)rxrzrRr}r{r6rrr r8as      z Mlp.forward) r!r"r#r$rGELUr r8r%rrrr rv;s rvcsFeZdZdZ   d fdd Zed fdd Zd d ZZ S) Attentionag Multi-head attention module with spatial awareness and trainable attention biases. This module implements a multi-head attention mechanism with support for spatial awareness, applying attention biases based on spatial resolution. It includes trainable attention biases for each unique offset between spatial positions in the resolution grid. Attributes: num_heads (int): Number of attention heads. scale (float): Scaling factor for attention scores. key_dim (int): Dimensionality of the keys and queries. nh_kd (int): Product of num_heads and key_dim. d (int): Dimensionality of the value vectors. dh (int): Product of d and num_heads. attn_ratio (float): Attention ratio affecting the dimensions of the value vectors. norm (nn.LayerNorm): Layer normalization applied to input. qkv (nn.Linear): Linear layer for computing query, key, and value projections. proj (nn.Linear): Linear layer for final projection. attention_biases (nn.Parameter): Learnable attention biases. attention_bias_idxs (Tensor): Indices for attention biases. ab (Tensor): Cached attention biases for inference, deleted during training. Methods: train: Sets the module in training mode and handles the 'ab' attribute. forward: Performs the forward pass of the attention mechanism. Examples: >>> attn = Attention(dim=256, key_dim=64, num_heads=8, resolution=(14, 14)) >>> x = torch.randn(1, 196, 256) >>> output = attn(x) >>> print(output.shape) torch.Size([1, 196, 256]) r(rcsttt|trt|dksJd||_|d|_||_|||_}t |||_ t ||||_ ||_ |j |d}t ||_t |||_t |j ||_ttt|dt|d}t|} i} g} |D],} |D]'} t| d| dt| d| df}|| vrt| | |<| | |qsqotj t|t| |_|jdt| | | ddd S) a  Initializes the Attention module for multi-head attention with spatial awareness. This module implements a multi-head attention mechanism with support for spatial awareness, applying attention biases based on spatial resolution. It includes trainable attention biases for each unique offset between spatial positions in the resolution grid. Args: dim (int): The dimensionality of the input and output. key_dim (int): The dimensionality of the keys and queries. num_heads (int): Number of attention heads. attn_ratio (float): Attention ratio, affecting the dimensions of the value vectors. resolution (Tuple[int, int]): Spatial resolution of the input feature map. Examples: >>> attn = Attention(dim=256, key_dim=64, num_heads=8, resolution=(14, 14)) >>> x = torch.randn(1, 196, 256) >>> output = attn(x) >>> print(output.shape) torch.Size([1, 196, 256]) r)z+'resolution' argument not tuple of length 2grrattention_bias_idxsF) persistentN) r r r`tuplerV num_headsscalekey_dimnh_kdr>ddh attn_ratiorrwrxryqkvprojra itertoolsproductrmabsappendr Parameterzerosattention_biasesregister_buffer LongTensorrW)rrPrrrr1rhpointsNattention_offsetsidxsp1p2offsetrrr r s4    ( "zAttention.__init__Tcs<t||rt|dr|`dS|jdd|jf|_dS)zTPerforms multi-head attention with spatial awareness and trainable attention biases.abN)r trainhasattrrrr)rmoderrr rs zAttention.trainc Cs|j\}}}||}||}||||jdj|j|j|jgdd\}}}|dddd}|dddd}|dddd}|j |j j |_ || dd|j|jr_|j dd|jfn|j } | jdd} | | dd|||j}||S) zSApplies multi-head attention with spatial awareness and trainable attention biases.rTr*)rPrr)rN)shaperxrrWrsplitrrrXrtordevicerZrtrainingrsoftmaxreshaperr) rr7r]r_rqkvattnrrr r8s   .  zAttention.forward)rr(r)T) r!r"r#r$r r no_gradrr8r%rrrr rks&:rcsFeZdZdZdddddejffdd Zdd Zd efd d Z Z S) TinyViTBlocka TinyViT Block that applies self-attention and a local convolution to the input. This block is a key component of the TinyViT architecture, combining self-attention mechanisms with local convolutions to process input features efficiently. Attributes: dim (int): The dimensionality of the input and output. input_resolution (Tuple[int, int]): Spatial resolution of the input feature map. num_heads (int): Number of attention heads. window_size (int): Size of the attention window. mlp_ratio (float): Ratio of MLP hidden dimension to embedding dimension. drop_path (nn.Module): Stochastic depth layer, identity function during inference. attn (Attention): Self-attention module. mlp (Mlp): Multi-layer perceptron module. local_conv (Conv2d_BN): Depth-wise local convolution layer. Methods: forward: Processes the input through the TinyViT block. extra_repr: Returns a string with extra information about the block's parameters. Examples: >>> input_tensor = torch.randn(1, 196, 192) >>> block = TinyViTBlock(dim=192, input_resolution=(14, 14), num_heads=3) >>> output = block(input_tensor) >>> print(output.shape) torch.Size([1, 196, 192]) r_r=r*c st||_||_||_|dksJd||_||_t|_ ||dks+Jd||} ||f} t || |d| d|_ t ||} | } t || | |d|_|d}t|||d||d|_d S) aH Initializes a TinyViT block with self-attention and local convolution. This block is a key component of the TinyViT architecture, combining self-attention mechanisms with local convolutions to process input features efficiently. Args: dim (int): Dimensionality of the input and output features. input_resolution (Tuple[int, int]): Spatial resolution of the input feature map (height, width). num_heads (int): Number of attention heads. window_size (int): Size of the attention window. Must be greater than 0. mlp_ratio (float): Ratio of MLP hidden dimension to embedding dimension. drop (float): Dropout rate. drop_path (float): Stochastic depth rate. local_conv_size (int): Kernel size of the local convolution. activation (torch.nn.Module): Activation function for MLP. Raises: AssertionError: If window_size is not greater than 0. AssertionError: If dim is not divisible by num_heads. Examples: >>> block = TinyViTBlock(dim=192, input_resolution=(14, 14), num_heads=3) >>> input_tensor = torch.randn(1, 196, 192) >>> output = block(input_tensor) >>> print(output.shape) torch.Size([1, 196, 192]) rz"window_size must be greater than 0z"dim must be divisible by num_headsr)rr1)r~rrr}r)r<N)r r rPrOr window_size mlp_ratiorrGrHrrr>rvmlpr local_conv)rrPrOrrrr}rHlocal_conv_sizer2head_dimwindow_resolutionmlp_hidden_dimmlp_activationrrrr r s" (  zTinyViTBlock.__init__c Cs|j\}}|j\}}}|||ksJd|}||jkr'||jkr'||}n|||||}|j||j|j}|j||j|j} |dkpL| dk} | r[t|ddd| d|f}|||| } } | |j} | |j}||| |j||j|dd|| ||j|j|}||}||| ||j|j|dd|| | |}| r|ddd|d|f }||||}|| |}|dd||||}| |}||||dd}|| | |S)zRApplies self-attention, local convolution, and MLP operations to the input tensor.zinput feature has wrong sizerr)r*Nr) rOrrrrWFrrZr contiguousrHrr)rr7rwrhwrres_xpad_bpad_rpaddingpHpWnHnWrrr r8Hs:      , zTinyViTBlock.forwardreturnc Cs,d|jd|jd|jd|jd|j S)a Returns a string representation of the TinyViTBlock's parameters. This method provides a formatted string containing key information about the TinyViTBlock, including its dimension, input resolution, number of attention heads, window size, and MLP ratio. Returns: (str): A formatted string containing the block's parameters. Examples: >>> block = TinyViTBlock(dim=192, input_resolution=(14, 14), num_heads=3, window_size=7, mlp_ratio=4.0) >>> print(block.extra_repr()) dim=192, input_resolution=(14, 14), num_heads=3, window_size=7, mlp_ratio=4.0 dim=, input_resolution=z , num_heads=z, window_size=z , mlp_ratio=)rPrOrrrrrrr extra_reprrs zTinyViTBlock.extra_repr r!r"r#r$rrr r8strrr%rrrr rs"A*rcsJeZdZdZddddddejdffdd Zd d Zd efd d Z Z S) BasicLayera A basic TinyViT layer for one stage in a TinyViT architecture. This class represents a single layer in the TinyViT model, consisting of multiple TinyViT blocks and an optional downsampling operation. Attributes: dim (int): The dimensionality of the input and output features. input_resolution (Tuple[int, int]): Spatial resolution of the input feature map. depth (int): Number of TinyViT blocks in this layer. use_checkpoint (bool): Whether to use gradient checkpointing to save memory. blocks (nn.ModuleList): List of TinyViT blocks that make up this layer. downsample (nn.Module | None): Downsample layer at the end of the layer, if specified. Methods: forward: Processes the input through the layer's blocks and optional downsampling. extra_repr: Returns a string with the layer's parameters for printing. Examples: >>> input_tensor = torch.randn(1, 3136, 192) >>> layer = BasicLayer(dim=192, input_resolution=(56, 56), depth=2, num_heads=3, window_size=7) >>> output = layer(input_tensor) >>> print(output.shape) torch.Size([1, 784, 384]) r_r=NFr*c sxt|_|_||_| |_tf ddt|D|_ | dur1d|_ dS| | d|_ dS)a Initializes a BasicLayer in the TinyViT architecture. This layer consists of multiple TinyViT blocks and an optional downsampling operation. It is designed to process feature maps at a specific resolution and dimensionality within the TinyViT model. Args: dim (int): Dimensionality of the input and output features. input_resolution (Tuple[int, int]): Spatial resolution of the input feature map (height, width). depth (int): Number of TinyViT blocks in this layer. num_heads (int): Number of attention heads in each TinyViT block. window_size (int): Size of the local window for attention computation. mlp_ratio (float): Ratio of MLP hidden dimension to embedding dimension. drop (float): Dropout rate. drop_path (float | List[float]): Stochastic depth rate. Can be a float or a list of floats for each block. downsample (nn.Module | None): Downsampling layer at the end of the layer. None to skip downsampling. use_checkpoint (bool): Whether to use gradient checkpointing to save memory. local_conv_size (int): Kernel size for the local convolution in each TinyViT block. activation (nn.Module): Activation function used in the MLP. out_dim (int | None): Output dimension after downsampling. None means it will be the same as `dim`. Raises: ValueError: If `drop_path` is a list and its length doesn't match `depth`. Examples: >>> layer = BasicLayer(dim=96, input_resolution=(56, 56), depth=2, num_heads=3, window_size=7) >>> x = torch.randn(1, 56 * 56, 96) >>> output = layer(x) >>> print(output.shape) c s8g|]}tttr|nd qS)) rPrOrrrr}rHrr2)rr`rarb r2rPr}rHrOrrrrrr rgs z'BasicLayer.__init__..Nrhri)rrPrOrjrrrr}rHrorkrr2rQrrr r s . zBasicLayer.__init__cCrp)zAProcesses input through TinyViT blocks and optional downsampling.Nrqrsrrr r8ruzBasicLayer.forwardrcCsd|jd|jd|jS)z:Returns a string with the layer's parameters for printing.rrz, depth=)rPrOrjrrrr rszBasicLayer.extra_reprrrrrr rs!MrcseZdZdZ         dfdd ZddZeddZej j ddZ ddZ ddZ ddgfddZZS)TinyViTa TinyViT: A compact vision transformer architecture for efficient image classification and feature extraction. This class implements the TinyViT model, which combines elements of vision transformers and convolutional neural networks for improved efficiency and performance on vision tasks. Attributes: img_size (int): Input image size. num_classes (int): Number of classification classes. depths (List[int]): Number of blocks in each stage. num_layers (int): Total number of layers in the network. mlp_ratio (float): Ratio of MLP hidden dimension to embedding dimension. patch_embed (PatchEmbed): Module for patch embedding. patches_resolution (Tuple[int, int]): Resolution of embedded patches. layers (nn.ModuleList): List of network layers. norm_head (nn.LayerNorm): Layer normalization for the classifier head. head (nn.Linear): Linear layer for final classification. neck (nn.Sequential): Neck module for feature refinement. Methods: set_layer_lr_decay: Sets layer-wise learning rate decay. _init_weights: Initializes weights for linear and normalization layers. no_weight_decay_keywords: Returns keywords for parameters that should not use weight decay. forward_features: Processes input through the feature extraction layers. forward: Performs a forward pass through the entire network. Examples: >>> model = TinyViT(img_size=224, num_classes=1000) >>> x = torch.randn(1, 3, 224, 224) >>> features = model.forward_features(x) >>> print(features.shape) torch.Size([1, 256, 64, 64]) r*`iir)r)r)r*r rrrrr_r=皙?F?c s t||_||_||_t||_||_tj }t ||d||d|_ |j j }||_ ddt d| t|D}t|_t|jD]|}t|||dd|dkrU|dn||dd|dkrd|dn|f|||t|d|t|d|d||jdkrtnd| |t|dt|d|d }|dkrtdd | i|}ntd|||||j| | d |}|j|qCt|d |_|dkrt|d |nt j|_||j| |t!tj"|d d dddt#d tj"d d ddddt#d |_$dS)a Initializes the TinyViT model. This constructor sets up the TinyViT architecture, including patch embedding, multiple layers of attention and convolution blocks, and a classification head. Args: img_size (int): Size of the input image. in_chans (int): Number of input channels. num_classes (int): Number of classes for classification. embed_dims (Tuple[int, int, int, int]): Embedding dimensions for each stage. depths (Tuple[int, int, int, int]): Number of blocks in each stage. num_heads (Tuple[int, int, int, int]): Number of attention heads in each stage. window_sizes (Tuple[int, int, int, int]): Window sizes for each stage. mlp_ratio (float): Ratio of MLP hidden dim to embedding dim. drop_rate (float): Dropout rate. drop_path_rate (float): Stochastic depth rate. use_checkpoint (bool): Whether to use checkpointing to save memory. mbconv_expand_ratio (float): Expansion ratio for MBConv layer. local_conv_size (int): Kernel size for local convolutions. layer_lr_decay (float): Layer-wise learning rate decay factor. Examples: >>> model = TinyViT(img_size=224, num_classes=1000) >>> x = torch.randn(1, 3, 224, 224) >>> output = model(x) >>> print(output.shape) torch.Size([1, 1000]) r)r-r.r1r2cSsg|]}|qSr)item)rcr7rrr rg\z$TinyViT.__init__..r)r*rN)rPrOrjrHrorkrQr2rf)rrrr}rrTF) kernel_sizer)rrrr)%r r r3 num_classesdepthsrV num_layersrrrr' patch_embedr+r linspacesumrllayersrmdictrKminr^rrrw norm_headryrGheadapply _init_weightsset_layer_lr_decayr/rrneck)rr3r-r embed_dimsrr window_sizesr drop_ratedrop_path_raterkmbconv_expand_ratiorlayer_lr_decayr2r+dpri_layerkwargslayerrrr r sx .   &$   zTinyViT.__init__cs|t|jfddtDdd|jfddd|jD](}|jD]}|fddd 7q,|jd urO|jfd dq'ksVJ|j|j fD] }|fd dq\| D]\}}||_ qmd d}||d S)zISets layer-wise learning rate decay for the TinyViT model based on depth.csg|] }|dqS)rrrb) decay_raterjrr rgsz.TinyViT.set_layer_lr_decay..cSs|D]}||_qdS)zTSets the learning rate scale for each layer in the model based on the layer's depth.N) parameterslr_scale)mrprrr _set_lr_scales z1TinyViT.set_layer_lr_decay.._set_lr_scalec|dS)Nrrr7r lr_scalesrr z,TinyViT.set_layer_lr_decay..rcs|S)Nrrrrdr rr r r rNcs|dS)Nrrrr rr r scr)NrTrrr rr r r cSs&|D] }t|dsJ|jqdS)zNChecks if the learning rate scale attribute is present in module's parameters.rN)rr param_name)rrrrr _check_lr_scales z3TinyViT.set_layer_lr_decay.._check_lr_scale) rrrmrrrrnrorrnamed_parametersr)rrrblockrrrrr)rrrjrdr r rs(      zTinyViT.set_layer_lr_decaycCsbt|tjr|jdurtj|jddSdSt|tjr/tj|jdtj|jddSdS)zMInitializes weights for linear and normalization layers in the TinyViT model.Nrr)r`rryrrrrwr)rrrr rs   zTinyViT._init_weightscCsdhS)zJReturns a set of keywords for parameters that should not use weight decay.rrrrrr no_weight_decay_keywordssz TinyViT.no_weight_decay_keywordscCs||}|jd|}d}t|t|jD] }|j|}||}q|j\}}}|||jdd|jdd|}|dddd}||S)zNProcesses input through feature extraction layers, returning spatial features.rrr(r*r)) rrrmrVrrWr+rXr)rr7start_irdrbatchrchannelrrr forward_featuress    $ zTinyViT.forward_featurescCr5)z^Performs the forward pass through the TinyViT model, extracting features from the input image.)rr6rrr r8r9zTinyViT.forwardicCsdd|D}||_t|jD]>\}}|dd|dkr|dn||dd|dkr.|dn|f}||_|jdur?||j_t|trM|jD]}||_qGqdS)zCSet image size to make model compatible with different image sizes.cSsg|]}|dqS)r(r)rcsrrr rgrz%TinyViT.set_imgsz..rr)r*rN)r+ enumeraterrOror`rrn)rimgszrdrrOrrrr set_imgszs   zTinyViT.set_imgsz)rr*rrrrrr_r=rFr_r*r)r!r"r#r$r r staticmethodrr jitignorerrr8rr%rrrr rs2$y#  r)rtypingrr torch.nnrtorch.nn.functional functionalrtorch.utils.checkpointutilsrrultralytics.nn.modulesrultralytics.utils.instancerr/rModuler'r;rKr^rvrrrrrrrr s&    &*=9]0~s