o 0ñ hW5ã@shddlmZddlmZddlZddlmmZddlmZGdd„deƒZ Gdd„dej ƒZ d ej d ej d e fd d „ZGdd„dejƒZGdd„dejƒZGdd„dejƒZGdd„dejƒZGdd„dejƒZGdd„dejƒZGdd„dejƒZGdd„dejƒZGdd„dejƒZGd d!„d!ejƒZGd"d#„d#ejƒZGd$d%„d%ejƒZGd&d'„d'ejƒZdS)(é)ÚAny)ÚBaseSparsifierN)ÚnncsReZdZdeeefddf‡fdd„ Zdejdedeeefddfdd „Z ‡Z S) ÚImplementedSparsifierÚkwargsÚreturnNc stƒj|ddS)N)Údefaults)ÚsuperÚ__init__)Úselfr©Ú __class__©úZ/var/www/vscode/kcb/lib/python3.10/site-packages/torch/testing/_internal/common_pruning.pyr szImplementedSparsifier.__init__ÚmoduleÚ tensor_namecKs4d|jjdjd<|jd}| dd¡d|d<dS)Nrzlinear1.weightÚ step_counté)ÚparametrizationsÚweightÚmaskÚstateÚget)r rrrÚ linear_staterrrÚ update_mask s z!ImplementedSparsifier.update_mask) Ú__name__Ú __module__Ú __qualname__ÚdictÚstrrr rÚModulerÚ __classcell__rrr rr s,rc@s(eZdZdZedejddfdd„ƒZdS)ÚMockSparseLinearzË This class is a MockSparseLinear class to check convert functionality. It is the same as a normal Linear layer, except with a different type, as well as an additional from_dense method. ÚmodrcCs||j|jƒ}|S)z )Ú in_featuresÚ out_features)Úclsr#ÚlinearrrrÚ from_densesÿzMockSparseLinear.from_denseN)rrrÚ__doc__Ú classmethodrÚLinearr(rrrrr"sr"Ú subset_tensorÚsuperset_tensorrcCsLd}|D]}|t|ƒkr t |||¡s|d7}nn |t|ƒks dSqdS)zW Checks to see if all rows in subset tensor are present in the superset tensor rrFT)ÚlenÚtorchÚequal)r,r-ÚiÚrowrrrÚrows_are_subset"s   üþr3có8eZdZdZd ‡fdd„ Zdejdejfdd„Z‡ZS) Ú SimpleLinearzžModel with only Linear layers without biases, some wrapped in a Sequential, some following the Sequential. Used to test basic pruned Linear-Linear fusion.rNc sftƒ ¡t tjddddtjddddtjdddd¡|_tjdddd|_tjdddd|_dS)NééF©Úbiasééé )r r rÚ Sequentialr+ÚseqÚlinear1Úlinear2©r r rrr 6s ýzSimpleLinear.__init__ÚxcCó"| |¡}| |¡}| |¡}|S©N)r>r?r@©r rBrrrÚforward@ó   zSimpleLinear.forward©rN© rrrr)r r/ÚTensorrFr!rrr rr52ó r5cr4) Ú LinearBiaszŠModel with only Linear layers, alternating layers with biases, wrapped in a Sequential. Used to test pruned Linear-Bias-Linear fusion.rNc s^tƒ ¡t tjddddtjddddtjddddtjddddtjdddd¡|_dS) Nr6r7Tr8r:Fér<)r r rr=r+r>rAr rrr Ks  ûzLinearBias.__init__rBcCs| |¡}|SrD)r>rErrrrFUs zLinearBias.forwardrHrIrrr rrLGrKrLcr4) ÚLinearActivationzùModel with only Linear layers, some with bias, some in a Sequential and some following. Activation functions modules in between each Linear in the Sequential, and each outside layer. Used to test pruned Linear(Bias)-Activation-Linear fusion.rNc s†tƒ ¡t tjddddt ¡tjddddt ¡tjdddd¡|_tjdddd|_t ¡|_ tjdd dd|_ t ¡|_ dS) Nr6r7Tr8r:Fr;rMr<) r r rr=r+ÚReLUÚTanhr>r?Úact1r@Úact2rAr rrr _s û zLinearActivation.__init__rBcCó6| |¡}| |¡}| |¡}| |¡}| |¡}|SrD)r>r?rQr@rRrErrrrFmó     zLinearActivation.forwardrHrIrrr rrNZsrNcr4) ÚLinearActivationFunctionala,Model with only Linear layers, some with bias, some in a Sequential and some following. Activation functions modules in between each Linear in the Sequential, and functional activationals are called in between each outside layer. Used to test pruned Linear(Bias)-Activation-Linear fusion.rNc sŽtƒ ¡t tjddddt ¡tjddddt ¡tjdddd¡|_tjdddd|_tjdd dd|_tjd d dd|_ t ¡|_ dS) Nr6r7Tr8r:Fr;rMér<) r r rr=r+rOr>r?r@Úlinear3rQrAr rrr |s ûz#LinearActivationFunctional.__init__rBcCsJ| |¡}| |¡}t |¡}| |¡}t |¡}| |¡}t |¡}|SrD)r>r?ÚFÚrelur@rWrErrrrFŠs       z"LinearActivationFunctional.forwardrHrIrrr rrUvsrUcr4) Ú SimpleConv2dz‡Model with only Conv2d layers, all without bias, some in a Sequential and some following. Used to test pruned Conv2d-Conv2d fusion.rNc shtƒ ¡t tjddddddtjdddddd¡|_tjdddddd|_tjdddddd|_dS) Nré rMFr8é@é0é4©r r rr=ÚConv2dr>Úconv2d1Úconv2d2rAr rrr ™s þzSimpleConv2d.__init__rBcCrCrD©r>rarbrErrrrF¢rGzSimpleConv2d.forwardrHrIrrr rrZ•s rZcr4) Ú Conv2dBiaszˆModel with only Conv2d layers, some with bias, some in a Sequential and some outside. Used to test pruned Conv2d-Bias-Conv2d fusion.rNc sztƒ ¡t tjddddddtjddddddtjdddddd¡|_tjdddddd|_tjdd dddd|_dS© Nrr[rMTr8r\Fr]r^r_rAr rrr ­s ýzConv2dBias.__init__rBcCrCrDrcrErrrrF·rGzConv2dBias.forwardrHrIrrr rrd©rKrdcr4) ÚConv2dActivationaModel with only Conv2d layers, some with bias, some in a Sequential and some following. Activation function modules in between each Sequential layer, functional activations called in-between each outside layer. Used to test pruned Conv2d-Bias-Activation-Conv2d fusion.rNc sŒtƒ ¡t tjddddddt ¡tjddddddt ¡tjddddddt ¡¡|_tjdddddd|_tjdd dddd|_ dSre) r r rr=r`rOrPr>rarbrAr rrr Äs úzConv2dActivation.__init__rBcCs6| |¡}| |¡}t |¡}| |¡}t |¡}|SrD)r>rarXrYrbÚhardtanhrErrrrFÑrTzConv2dActivation.forwardrHrIrrr rrf¾s rfcr4) Ú Conv2dPadBiasaQModel with only Conv2d layers, all with bias and some with padding > 0, some in a Sequential and some following. Activation function modules in between each layer. Used to test that bias is propagated correctly in the special case of pruned Conv2d-Bias-(Activation)Conv2d fusion, when the second Conv2d layer has padding > 0.rNcsÚtƒ ¡t tjdddddddt ¡tjddddddt ¡tjdddddddt ¡tjdddddddt ¡tjddddddt ¡¡ |_tjdd ddddd|_t ¡|_ tjd d ddddd|_ t ¡|_ dS) Nrr[rMT)Úpaddingr9Fr8r\r]r^) r r rr=r`rOrPr>rarQrbrRrAr rrr às" ö zConv2dPadBias.__init__rBcCrSrD)r>rarQrbrRrErrrrFórTzConv2dPadBias.forwardrHrIrrr rrhÚsrhcr4) Ú Conv2dPoolzèModel with only Conv2d layers, all with bias, some in a Sequential and some following. Activation function modules in between each layer, Pool2d modules in between each layer. Used to test pruned Conv2d-Pool2d-Conv2d fusion.rNc sÂtƒ ¡t tjddddddtjddddt ¡tjddddddt ¡tjdddd¡|_ tjdd dddd|_ tjdddd|_ t ¡|_ tjd d dddd|_ tjd d dddd|_dS) Nrr[rMT©Ú kernel_sizerir9é©rlÚstriderir\r]r^)r r rr=r`Ú MaxPool2drOrPÚ AvgPool2dr>raÚmaxpoolÚaf1rbÚconv2d3rAr rrr s ú zConv2dPool.__init__rBcCs\| |¡}| |¡}| |¡}| |¡}| |¡}tj|dddd}t |¡}| |¡}|S©Nrmrrn) r>rarrrsrbrXÚ avg_pool2drYrtrErrrrFs       zConv2dPool.forwardrHrIrrr rrjüsrjcr4) ÚConv2dPoolFlattenFunctionalaModel with Conv2d layers, all with bias, some in a Sequential and some following, and then a Pool2d and a functional Flatten followed by a Linear layer. Activation functions and Pool2ds in between each layer also. Used to test pruned Conv2d-Pool2d-Flatten-Linear fusion.rNc s¸tƒ ¡t tjddddddtjddddt ¡tjddddddt ¡tjdddd¡|_ tjdddddd|_ t ¡|_ tjdd dddd|_ t  d ¡|_tjd d dd |_dS) NrrMTrkrmrnr7r6é )rré r8)r r rr=r`rprOrPrqr>rarsrbÚAdaptiveAvgPool2dÚavg_poolr+ÚfcrAr rrr #s ú  z$Conv2dPoolFlattenFunctional.__init__rBcCs^| |¡}| |¡}tj|dddd}| |¡}| |¡}| |¡}t |d¡}|  |¡}|Sru) r>rarXÚ max_pool2drsrbr{r/Úflattenr|rErrrrF3s       z#Conv2dPoolFlattenFunctional.forwardrHrIrrr rrwsrwcr4) ÚConv2dPoolFlattenaModel with Conv2d layers, all with bias, some in a Sequential and some following, and then a Pool2d and a Flatten module followed by a Linear layer. Activation functions and Pool2ds in between each layer also. Used to test pruned Conv2d-Pool2d-Flatten-Linear fusion.rNc sÂtƒ ¡t tjddddddtjddddt ¡tjddddddt ¡tjdddd¡|_ tjdddddd|_ t ¡|_ tjdd dddd|_ t  d ¡|_t ¡|_tjd d dd |_dS)NrrMTrkrmrnr7r6rx)rmrmé,ryr8)r r rr=r`rprOrPrqr>rarsrbrzr{ÚFlattenr~r+r|rAr rrr Es ú   zConv2dPoolFlatten.__init__rBcCs\| |¡}| |¡}tj|dddd}| |¡}| |¡}| |¡}| |¡}| |¡}|Sru) r>rarXr}rsrbr{r~r|rErrrrFVs       zConv2dPoolFlatten.forwardrHrIrrr rr?src óVeZdZdZdededededdf ‡fdd „ Zd ejdeejejffd d „Z ‡Z S) ÚLSTMLinearModelzCContainer module with an encoder, a recurrent module, and a linear.Ú input_dimÚ hidden_dimÚ output_dimÚ num_layersrNcs,tƒ ¡t |||¡|_t ||¡|_dSrD)r r rÚLSTMÚlstmr+r'©r r„r…r†r‡r rrr es zLSTMLinearModel.__init__ÚinputcCs | |¡\}}| |¡}||fSrD)r‰r')r r‹ÚoutputÚ_hiddenÚdecodedrrrrFls zLSTMLinearModel.forward© rrrr)Úintr r/rJÚtuplerFr!rrr rrƒbsÿÿÿÿþ(rƒc r‚) ÚLSTMLayerNormLinearModelz9Container module with an LSTM, a LayerNorm, and a linear.r„r…r†r‡rNcs8tƒ ¡t |||¡|_t |¡|_t ||¡|_dSrD) r r rrˆr‰Ú LayerNormÚnormr+r'rŠr rrr us  z!LSTMLayerNormLinearModel.__init__rBcCs*| |¡\}}| |¡}| |¡}||fSrD)r‰r”r')r rBrrrrrF}s  z LSTMLayerNormLinearModel.forwardrrrr rr’rsÿÿÿÿþ(r’)ÚtypingrÚtorch.ao.pruningrr/Útorch.nn.functionalrÚ functionalrXrr+r"rJÚboolr3r r5rLrNrUrZrdrfrhrjrwrrƒr’rrrrÚs*    "!"#