o I hG-@sddlmZddlZgdZGdddejjZGdddejjZGdd d ejjZGd d d ejj Z Gd d d ejj Z Gdddejj Z Gdddej jj jZGdddejjZdS))warnN)ReLU6 HardswishELU LeakyReLUSigmoidSoftmaxMultiheadAttentionPReLUcs@eZdZdZd fdd ZddZddZed d d ZZ S) raApplies the element-wise function: :math:`\text{ReLU6}(x) = \min(\max(x_0, x), q(6))`, where :math:`x_0` is the zero_point, and :math:`q(6)` is the quantized representation of number 6. Args: inplace: can optionally do the operation in-place. Default: ``False`` Shape: - Input: :math:`(N, *)` where `*` means, any number of additional dimensions - Output: :math:`(N, *)`, same shape as the input .. image:: ../scripts/activation_images/ReLU6.png Examples:: >>> m = nn.quantized.ReLU6() >>> input = torch.randn(2) >>> # xdoctest: +SKIP >>> input = torch.quantize_per_tensor(input, 1.0, 0, dtype=torch.qint32) >>> output = m(input) Fcst|||_dSN)super__init__inplace)selfr __class__\/var/www/vscode/kcb/lib/python3.10/site-packages/torch/ao/nn/quantized/modules/activation.pyr ,s  zReLU6.__init__cCstjj||jSr )torchops quantizedrelu6rrinputrrrforward0sz ReLU6.forwardcCdS)NQuantizedReLU6rrrrr _get_name3zReLU6._get_namecCs t|jSr )rr)moduse_precomputed_fake_quantrrr from_float6s zReLU6.from_floatF) __name__ __module__ __qualname____doc__r rr staticmethodr" __classcell__rrrrrsrcLeZdZdZdfdd ZddZddZedd d Ze d d Z Z S)rzThis is the quantized version of :class:`~torch.nn.Hardswish`. Args: scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor NcsL||d}t|dtj|fi||dtj|fi|dSN)devicedtypescale zero_pointr r register_bufferrtensor)rr.r/r,r-factory_kwargsrrrr Cs  zHardswish.__init__cCtjj||j|jSr )rrr hardswishr.r/rrrrrIszHardswish.forwardcCr)NQuantizedHardswishrrrrrrLrzHardswish._get_nameFcCs |j\}}tt|t|Sr )activation_post_processcalculate_qparamsrfloatintr r!r.r/rrrr"OszHardswish.from_floatcCs|t|t|Sr )r9r:clsr r.r/rrrfrom_referenceTszHardswish.from_reference)NNr# r$r%r&r'r rrr(r" classmethodr>r)rrrrr;s rcr*)rzThis is the quantized equivalent of :class:`~torch.nn.ELU`. Args: scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor alpha: the alpha constant ?cst|||_||_dSr )r r r.r/)rr.r/alpharrrr bs  z ELU.__init__cCstjjjj||j|j|jSr ) raonnr functionalelur.r/rBrrrrrgs z ELU.forwardcCr)N QuantizedELUrrrrrrlrz ELU._get_nameFcCs$|j\}}tt|t||jSr )r7r8rr9r:rBr;rrrr"ozELU.from_floatcCs|t|t||jSr )r9r:rBr<rrrr>tzELU.from_reference)rAr#r?rrrrrYs rc sjeZdZdZ    ddedededed df fd d Zd d ZddZ e dddZ e ddZ Z S)raThis is the quantized equivalent of :class:`~torch.nn.LeakyReLU`. Args: scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor negative_slope: Controls the angle of the negative slope. Default: 1e-2 {Gz?FNr.r/negative_sloperreturncsP||d}t|||dtj|fi||dtj|fi|dSr+r0)rr.r/rKrr,r-r3rrrr s zLeakyReLU.__init__cCstjj||j|j|j|jSr )rrr leaky_relurKrr.r/rrrrrszLeakyReLU.forwardcCr)NQuantizedLeakyReLUrrrrrrrzLeakyReLU._get_namecCs(|j\}}|t|t||j|jSr )r7r8r9r:rKr)r=r r!r.r/rrrr"szLeakyReLU.from_floatcCs|t|t||j|jSr )r9r:rKrr<rrrr>szLeakyReLU.from_reference)rJFNNr#)r$r%r&r'r9r:boolr rrr@r"r>r)rrrrrys.  rcs@eZdZdZdedeffdd ZddZed d d Z Z S) rzThis is the quantized equivalent of :class:`~torch.nn.Sigmoid`. Args: scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor output_scaleoutput_zero_pointcst||_||_dSr )r r rPrQ)rrPrQrrrr s  zSigmoid.__init__cCr4r )rrrsigmoidrPrQrrrrrs zSigmoid.forwardFcCs |j\}}|t|t|Sr )r7r8r9r:)r=r r!rPrQrrrr"s zSigmoid.from_floatr#) r$r%r&r'r9r:r rr@r"r)rrrrrs rcsLeZdZdZdfdd ZddZd d Zedd d Ze ddZ Z S)ra,This is the quantized version of :class:`~torch.nn.Softmax`. Args: dim: A dimension along which Softmax will be computed (so every slice along dim will sum to 1). scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor NrArcs t||_||_||_dSr )r r dimr.r/)rrSr.r/rrrr s  zSoftmax.__init__cCs@|j}|durd}tjjd||}tjj|||j|j S)Nsoftmax) rSrrDrE_get_softmax_dimrrrUr.r/)rrrS stacklevelrrrrs zSoftmax.forwardcCr)NQuantizedSoftmaxrrrrrrrzSoftmax._get_nameFcCs$|j\}}t|jt|t|Sr )r7r8rrSr9r:r;rrrr"rHzSoftmax.from_floatcCs||jt|t|Sr )rSr9r:r<rrrr>rIzSoftmax.from_reference)NrArr#r?rrrrrs  rc@s8eZdZejjjjZddZ e ddZ e ddZ dS)r cCr)NQuantizedMultiheadAttentionrrrrrrrzMultiheadAttention._get_namecCstd)NzpIt looks like you are trying to convert a non-observed MHA module. Please, see the examples on quantizable MHAs.)NotImplementedError)r=otherrrrr"szMultiheadAttention.from_floatcCstjjj|ddddd}||_|jdur2|jd}tj|dd\}}t |||tj }t |d||j durU|jd}tj|dd\}}t |||tj }t |d||` |`|S)NFT)mappingrremove_qconfigconvert_custom_config_dictbias_k) reduce_rangebias_v)rrC quantizationconvertrr_ _parameterspop_choose_qparams_per_tensorquantize_per_tensorquint8setattrrain_proj_weight in_proj_bias)r=r[ convertedr_sczprarrr from_observeds.       z MultiheadAttention.from_observedN) r$r%r&rrCrD quantizabler _FLOAT_MODULErr@r"rorrrrr s   r c seZdZdZ ddedededdffdd Zd ejddfd d Z d ejdejfddZ ddZ e dddZ e ddZZS)r a%This is the quantized equivalent of :class:`~torch.nn.PReLU`. Args: scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor num_parameters: number of parameters: 1, or the number of channels at input. Default: 1 rPrQnum_parametersrLNcsNt||_||_||_tj|tjd}tj|ddtj d}| |dS)N)r-rAr)r.r/r-) r r rsr.r/rrandnr9rgrh set_weight)rrPrQrswqwrrrr s zPReLU.__init__rvcCs ||_dSr )weight)rrvrrrru+s zPReLU.set_weightrcCstjj||j|j|jSr )rrrprelurxr.r/rrrrr.sz PReLU.forwardcCr)NQuantizedPReLUrrrrrr3rzPReLU._get_nameFc Cs|j\}}|t|t||j}|j}|j}|||jtj kr.t d|j|\}} t |t|t| tj } | | |SNz9PReLU's weight observer should have dtype quint8 but got ) r7r8r9r:rsrxqconfigr-rrhrrgru) r=r r!r.r/qprelufloat_wtobserverwt_scalewt_zpqweightrrrr"6s      zPReLU.from_floatc Cs|t|t||j}|j}|j}|||jtjkr'td|j| \}}t |t|t|tj} | | |Sr{) r9r:rsrxr|r-rrhrr8rgru) r=r r.r/r}r~rrrrrrrr>Hs      zPReLU.from_reference)rrr#)r$r%r&r'r9r:r rTensorrurrr@r"r>r)rrrrr s&   r )warningsrr__all__rDReLUrrrrrrrCrpr Moduler rrrrs  ( )'3