o I h@s$ ddlZddlmZddlZddlmZddlmZmZddl m Z m Z e ddZ ej ejejejejgZejejgZdd eDZed d eDd d Ze d e e dddejdededededejdejfddZe e dddejdededededejdejfddZe de e dddejdejdejdededejdejfddZ e e dddejdejdejdededejdejfdd Z!e d!e e d"ddejdejdejdejdejdejdejfd#d$Z"e e d"ddejdejdejdejdejdejdejfd%d&Z#e d'e e d(ddd)dejdededededejd*eejdejfd+d(Z$e e d(ddd)dejdejdejdededejd*eejdejfd,d-Z%e d.e e d/ddd)dejdejdejdededejd*eejdejfd0d1Z&e e d/ddd)dejdejdejdededejd*eejdejfd2d3Z'e d4e e d5ddd)dejdejdejdejdejdejd*eejdejfd6d7Z(e e d5ddd)d*eejdejfd8d9Z)e d:e e d;ddejdedejde*ejejff d?d@Z+e dAe e dBddejdedejde*ejejff dCdDZ,e e d;ddejdeded>edejde*ejejff dEdFZ-e e dBddejdeded>edejde*ejejff dGdHZ.dIdJZ/e dKe e dLddejdMejdNejdOedededejdejfdPdLZ0e e dLddejdMejdNejdOedededejdejfdQdRZ1e dSe e dTddd)dejdMejdNeejdOedededejd*eejdejfdUdTZ2e e dTddd)dejdMejdNeejdOedededejd*eejdejfdVdWZ3e dXe e dYddejdejde*ejejffdZdYZ4e e dYddejdejde*ejejffd[d\Z5e d]e e d^d_dejdejde*ejejffd`d^Z6e dae e dbddejdejde*ejejffdcdbZ7e e dbddejdejde*ejejffdddeZ8dfdgZ9e dhe e diddejdMejdNejdededejf djdiZ:e e diddejdMejdNejdededejf dkdlZ;e dme e dndeje dse e dtd uddejdMejdNejdededejf dvdtZ?e e dtd uddejdMejdNejdededejf dwdxZ@e dye e dzdduejsrcCs.i|]}|tt|jtt|jfqSr )intr finfor rrr r rrs.cCs^|tvr td|t|\}}||ksJd|d|||ks-Jd|d|dS)NzUnsupported dtype: z9quant_min out of bound for dtype, quant_min_lower_bound: z quant_min: z9quant_max out of bound for dtype, quant_max_upper_bound: z quant_max: )_DTYPE_TO_QVALUE_BOUNDS ValueError) quant_min quant_maxdtypequant_min_lower_boundquant_max_upper_boundr r r_quant_min_max_bounds_checks"   rzxquantize_per_tensor(Tensor input, float scale, int zero_point, int quant_min, int quant_max, ScalarType dtype) -> Tensorquantize_per_tensorCompositeExplicitAutogradinputscale zero_pointrrrreturncCsp|jtjtjfvr|tj}|jtjksJd|jt|||d|}tt||||||S)aAffine quantization for the Tensor using the same quantization parameters to map from floating point to quantized values Args: input (torch.Tensor): original float32 or bfloat16 Tensor scale (float): quantization parameter for affine quantization zero_point (int): quantization parameter for affine quantization quant_min (int): minimum quantized value for output Tensor quant_max (int): maximum quantized value for output Tensor dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor Returns: Tensor with requested dtype (e.g. torch.uint8), note the quantization parameters are not stored in the Tensor, we are storing them in function arguments instead Tensorzquantize_per_tensor.tensorcCsV|dksJd||dksJd|t||||||SzAffine quantization for the Tensor using the same quantization parameters to map from floating point to quantized values Same as `quantize_per_tensor` but scale and zero_point are Scalar Tensor instead of scalar values >Expecting zero_point tensor to be one element, but received : 9Expecting scale tensor to be one element, but received : numelritemr0r r rquantize_per_tensor_tensorms  r9cCs|jtjtjfvr|tj}|dksJd||dks-Jd||jtjks;Jd|jtj||dS)Nr3r4r5r$r.)rr r&r'r(r)r7r/r0r r rquantize_per_tensor_tensor_metas    r:zquantize_per_tensor.tensor2(Tensor input, Tensor scale, Tensor zero_point, Tensor quant_min, Tensor quant_max, ScalarType dtype) -> Tensorzquantize_per_tensor.tensor2cCs^|dksJd||dksJd|t||||||Sr2r6r0r r rquantize_per_tensor_tensor2s  r;cCst||||||SN)r:r0r r r quantize_per_tensor_tensor2_metas r=zdequantize_per_tensor(Tensor input, float scale, int zero_point, int quant_min, int quant_max, ScalarType dtype, *, ScalarType? out_dtype=None) -> Tensordequantize_per_tensor out_dtyper@cCsV|j|ksJd|d|j|durtj}|tvr$||||Std|)aAffine dequantization for the Tensor using the same quantization parameters to map from quantized values to floating point values Args: input (torch.Tensor): Tensor with dtype matching `dtype` argument, e.g. (`torch.uint8`), it is a per tensor quantized Tensor if combined with quantization parameters in the argument of this function (scale/zero_point) scale (float): quantization parameter for affine quantization zero_point (int): quantization parameter for affine quantization quant_min (int): minimum quantized value for input Tensor (not used in computation, reserved for pattern matching) quant_max (int): maximum quantized value for input Tensor (not used in computation, reserved for pattern matching) dtype (torch.dtype): dtype for input Tensor (not used in computation, reserved for pattern matching) out_dtype (torch.dtype?): optional dtype for output Tensor Returns: dequantized float32 Tensor Expecting input to have dtype: z , but got N,Unsupported dtype in dequantize_per_tensor: )rr r)rr(rr r!r"rrrr@r r rr>s &cCs|durtj}tj||dSNr.)r r)r/rCr r rdequantize_per_tensor_metas rEzdequantize_per_tensor.tensor(Tensor input, Tensor scale, Tensor zero_point, int quant_min, int quant_max, ScalarType dtype, *, ScalarType? out_dtype=None) -> Tensorzdequantize_per_tensor.tensorc CsZ|dksJd||dksJd|t|||||||dSzAffine dequantization for the Tensor using the same quantization parameters to map from quantized values to floating point values Same as `dequantize_per_tensor` but scale and zero_point are Scalar Tensor instead of scalar values r3r4r5r?r7r>r8rCr r rdequantize_per_tensor_tensor's  rHcCs|durtj}|dksJd||dks%Jd||j|ks1Jd||tvr Tensorzdequantize_per_tensor.tensor2c Csb|dksJd||dksJd|t|||||||dSrFrGrCr r rdequantize_per_tensor_tensor2ms  rJc Cst|||||||dS)Nr?)rIrCr r r"dequantize_per_tensor_tensor2_metas rKzrchoose_qparams.tensor(Tensor input, int quant_min, int quant_max, float eps, ScalarType dtype) -> (Tensor, Tensor)zchoose_qparams.tensorqminqmaxepsc Cs||jtjtjtjfvsJd|j|tvs#Jdtd|t||t|\}}t |||||t |gddS)[Given an input Tensor, derive the per tensor affine quantization parameter (scale and zero_point) for target quantized Tensor from the Tensor Args: input (torch.Tensor): floating point input Tensor quant_min (int): minimum quantized value for target quantized Tensor quant_max (int): maximum quantized value for target quantized Tensor dtype (torch.dtype): dtype for target quantized Tensor Returns: scale (float): quantization parameter for the target quantized Tensor zero_point (int): quantization parameter for the target quantized Tensor CExpecting input to have dtype torch.float32/16/b16, but got dtype: $Expecting target dtype to be one of , but got: F)has_customized_qrange) rr r)r&r'rkeysraminmaxrTensorr rLrMrNrmin_valmax_valr r rchoose_qparams_tensors*    rZz|choose_qparams_symmetric.tensor(Tensor input, int quant_min, int quant_max, float eps, ScalarType dtype) -> (Tensor, Tensor)zchoose_qparams_symmetric.tensorc Cs|jtjtjtjfvsJd|j|tvs#Jdtd|t||t|\}}t |||||t |gdtj dS)rOrPrQrRF)rSqscheme) rr r)r&r'rrTrrUrrVper_tensor_symmetricrWr r rchoose_qparams_symmetric_tensors,    r]cCsj|jtjtjtjfvsJd|j||ks!Jd|d|tjdtj|jdtjdtj|jdfS)NrPzKExpecting quant_min to be smaller than quant_max but received min: z max: r3rdevice) rr r)r&r'emptydoubler_int64r rrrNrr r rchoose_qparams_tensor_metas"   rdcCs(tjdtj|jdtjdtj|jdfS)Nr3r^)r r`rar_rbrcr r r$choose_qparams_symmetric_tensor_metas recCs6tt|}d||<||d<|t|}||fS)Nr)listrangedimpermutetuple)xaxis new_axis_listyr r r_permute_to_axis_zeros rozquantize_per_channel(Tensor input, Tensor scales, Tensor zero_points, int axis, int quant_min, int quant_max, ScalarType dtype) -> Tensorquantize_per_channelscales zero_pointsrlc Cs|jtjtjfvr|tj}|jtjksJd|j||ks,Jd|t|||t||\}}dg|}|j d|d<| |}| |}t t |d||||} | t|} | |S)atAffine per channel quantization for the Tensor using the same quantization parameters for each channel/axis to map from floating point to quantized values Args: input (torch.Tensor): original float32 or bfloat16 Tensor scales (torch.Tensor): a list of scale quantization parameter for affine quantization, one per channel zero_point (torch.Tensor): a list of zero_point quantization parameter for affine quantization, one per channel quant_min (int): minimum quantized value for output Tensor quant_max (int): maximum quantized value for output Tensor dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor Returns: Tensor with requested dtype (e.g. torch.uint8), note the quantization parameters are not stored in the Tensor, we are storing them in function arguments instead r$Expecting axis to be < r3rr%)rr r&r'r(r)rhrroshapeviewr*r+rirj) r rqrrrlrrrpermute_axis_list new_shaperesoutr r rrp,s"      cCsr|jtjtjfvr|tj}|jtjksJd|j||ks,Jd|t|||tj||dS)Nr$rsr.) rr r&r'r(r)rhrr/)r rqrrrlrrrr r rquantize_per_channel_meta\s   rzzdequantize_per_channel(Tensor input, Tensor scales, Tensor? zero_points, int axis, int quant_min, int quant_max, ScalarType dtype, *, ScalarType? out_dtype=None) -> Tensordequantize_per_channelc Cs|j|ksJd|d|j|durtj}||ks&Jd|t|||t||\}}dg|} |jd| d<|| }|durT||| |} n||} | |} | t |} | S)aAffine per channel dequantization for the Tensor using the same quantization parameters for each channel/axis to map from quantized values to floating point values Args: input (torch.Tensor): Tensor with dtype matching `dtype` argument, e.g. (`torch.uint8`), it is a per channel quantized Tensor if combined with quantization parameter in the argument of this function (scales/zero_points/axis) scales (torch.Tensor): a list of scale quantization parameter for affine quantization, one per channel zero_points (torch.Tensor): a list of zero_point quantization parameter for affine quantization, one per channel quant_min (int): minimum quantized value for output Tensor (not used in computation, reserved for pattern matching) quant_max (int): maximum quantized value for output Tensor (not used in computation, reserved for pattern matching) dtype (torch.dtype): requested dtype for output Tensor (not used in computation, reserved for pattern matching) out_dtype (torch.dtype?): optional dtype for output Tensor Returns: dequantized float32 Tensor Expecting input to have dtype , but got dtype: Nrsr3r) rr r)rhrrortrur(rirj) r rqrrrlrrrr@rvrwrxryr r rr{zs" )   cCsf|j|ksJd|d|j|durtj}||ks&Jd|t|||tj||dS)Nr|r}rsr.)rr r)rhrr/)r rqrrrlrrrr@r r rdequantize_per_channel_metas  r~zLchoose_qparams_per_token(Tensor input, ScalarType dtype) -> (Tensor, Tensor)choose_qparams_per_tokencCsx|jddd}|jtjkr|}|tjkr#d}d|dd}ntd||jdd  |}t |}||fS) Choose quantization parameters for per token quantization. This means for a N dimension Tensor (M1, M2, ...Mn, N), we calculate scales/zero_points for each N elements and quantize every N elements with the same quantization parameter. The dimension for scales/zero_points will be (M1 * M2 ... * Mn) Args: input (torch.Tensor): original float32/float16 Tensor dtype (torch.dtype): dtype (e.g. torch.uint8) for input Tensor Returns: scales and zero_points, both float32 Tensors Trhkeepdimr3z/unsupported dtype in choose_qparams_per_token: gh㈵>r ) absamaxrr r&floatint8 Exceptionr*div zeros_like)r rrqn_bitsrrrr r rrs   cC@t|jdddg}tj|tj|jdtj|tj|jdfSNrr3r^rfrtr r`rar_rbr rsizer r rchoose_qparams_per_token_meta  rz]_choose_qparams_per_token_asymmetric_impl(Tensor input, ScalarType dtype) -> (Tensor, Tensor))_choose_qparams_per_token_asymmetric_implCompositeImplicitAutogradcCsd\}}tj|ddd}tj|ddd}t|t|}t|t|}ttjj}||t ||} | j |d} || } || } || } || } t | | dk|| || }t ||| }| tj| tjfS)r)irTrrr)r aminrr rrrr)rNrr*wherer+r(float64rb)r rrLrMrXrY min_val_neg max_val_posrNr! descaled_min descaled_maxzero_point_from_min_errorzero_point_from_max_errorr"r r rrs&  zWchoose_qparams_per_token_asymmetric(Tensor input, ScalarType dtype) -> (Tensor, Tensor)#choose_qparams_per_token_asymmetriccCs t||Sr<)rr rr r rrE cCrrrrr r r(choose_qparams_per_token_asymmetric_metaQrrcCsftt|dd}||ksJd|d|||ks1Jd|d|dS)Nrz num_tokens: z scales: z zero_points: )mathprodrfrr7)r rqrr num_tokensr r r!_per_token_quant_qparam_dim_check`srz}quantize_per_token(Tensor input, Tensor scales, Tensor zero_points, int quant_min, int quant_max, ScalarType dtype) -> Tensorquantize_per_tokencCsBt|||t||||d|||||}|S)aPer token quantization for the Tensor using the quantization parameters to map from floating point to quantized values. This means for a N dimension Tensor (M1, M2, ...Mn, N), we calculate scales/zero_points for each N elements and quantize every N elements with the same quantization parameter. The dimension for scales/zero_points will be (M1 * M2 ... * Mn) Args: input (torch.Tensor): original float32 or bfloat16 Tensor scales (float32 torch.Tensor): quantization parameter for per token affine quantization zero_points (int32 torch.Tensor): quantization parameter for per token affine quantization quant_min (int): minimum quantized value for output Tensor quant_max (int): maximum quantized value for output Tensor dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor Returns: Tensor with requested dtype (e.g. torch.uint8), note the quantization parameters are not stored in the Tensor, we are storing them in function arguments instead r%)rrmuladdr+r*r(r rqrrrrrr r rrps   cCst|||tj||dSrDrr r/rr r rquantize_per_token_metas rzdequantize_per_token(Tensor input, Tensor scales, Tensor zero_points, int quant_min, int quant_max, ScalarType dtype, ScalarType output_dtype) -> Tensordequantize_per_token output_dtypecCs||}||}||S)aPer token dequantization for the Tensor using the quantization parameters to map from floating point to quantized values. This means for a N dimension Tensor (M1, M2, ...Mn, N), we calculate scales/zero_points for each N elements and quantize every N elements with the same quantization parameter. The dimension for scales/zero_points will be (M1 * M2 ... * Mn) Args: input (torch.Tensor): quantized Tensor (uint8, int8 etc.) scales (float64 torch.Tensor): quantization parameter for per token affine quantization zero_points (int64 torch.Tensor): quantization parameter for per token affine quantization quant_min (int): minimum quantized value for input Tensor quant_max (int): maximum quantized value for input Tensor dtype (torch.dtype): dtype (e.g. torch.uint8) for input Tensor output_dtype (torch.dtype): dtype (e.g. torch.float32) for output Tensor Returns: dequantized Tensor with dtype `output_dtype` )r(r rqrrrrrrr r rrs cCst|||tj||dSrDrrr r rdequantize_per_token_metas rzquantize_per_channel_group(Tensor input, Tensor scales, Tensor zero_points, int quant_min, int quant_max, ScalarType dtype, int group_size) -> Tensorquantize_per_channel_groupc Cs|dksJ||jdkr|jddkr|jd}|jd|dks$J|dks,J|d|}t|dks=J|dd}|dd}|d|| || | |}|S)Nr3rrrr%) rtrhreshaper isnansumrrr+clamp_r( reshape_as) r rqrrrrr group_sizeto_quant input_int8r r rrs"       cCsf|dksJ||jdkr|jddkr|jd}|jd|dks$J|dks,Jtj||dS)aXGroupwise quantization within each channel for an 2-d Tensor using the quantization parameters to map from floating point to quantized values. This means for each row of a 2-d Tensor (M, N), we calculate scales/zero_points for each `group_size` elements and quantize every `group_size` elements with the same quantization parameter. The dimension for scales/zero_points will be (M * ceil(N, group_size),) Args: input (torch.Tensor): original float32 or bfloat16 Tensor scales (float32 torch.Tensor): quantization parameter for per channel group affine quantization zero_points (int32 torch.Tensor): quantization parameter for per channel group affine quantization quant_min (int): minimum quantized value for output Tensor quant_max (int): maximum quantized value for output Tensor dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor Returns: Tensor with requested dtype (e.g. torch.uint8), note the quantization parameters are not stored in the Tensor, we are storing them in function arguments instead r3rrrr.)rtrhr r/)r rqrrrrrrr r rquantize_per_channel_group_meta s  rzdequantize_per_channel_group(Tensor input, Tensor scales, Tensor? zero_points, int quant_min, int quant_max, ScalarType dtype, int group_size, ScalarType output_dtype) -> Tensordequantize_per_channel_groupw_int8rc Cs|dksJ||jdkr|jddkr|jd}|jd|dks$J|dks,J|d|}|dd}|durC|dd} n tjgtj|jd} || | | |} | S)a!Groupwise dequantization within each channel for an 2-d Tensor using the quantization parameters to map from floating point to quantized values. This means for each row of a 2-d Tensor (M, N), we calculate scales/zero_points for each `group_size` elements and quantize every `group_size` elements with the same quantization parameter. The dimension for scales/zero_points will be (M * ceil(N, group_size),) Args: input (torch.Tensor): quantized Tensor (uint8/int8 etc.) scales (float32 torch.Tensor): quantization parameter for per channel group affine quantization zero_points (int32 torch.Tensor): quantization parameter for per channel group affine quantization quant_min (int): minimum quantized value for input Tensor quant_max (int): maximum quantized value for input Tensor dtype (torch.dtype): dtype (e.g. torch.uint8) for input Tensor output_dtype (torch.dtype): dtype (e.g. torch.float32) for output Tensor Returns: dequantized Tensor with dtype `output_dtype` r3rrrNr^) rtrhrr zerosint32r_subrrr() rrqrrrrrrrw_int8_groupedzpw_dqr r rr5s "   zyfake_quant_per_channel(Tensor input, Tensor scales, Tensor zero_points, int axis, int quant_min, int quant_max) -> Tensorc@s$eZdZeddZeddZdS)FakeQuantPerChannelc Cs|jtjkr |tj}|jtjkr|tj}|jtjks&Jd|j||ks5Jd|ttd|tt|d|j}t ||}t ||} t |d|| } t | ||| |} t | |k| |k} | | | S)Nr$rsrr3r%)rr r)r(rrhrfrgndimrr+r* logical_andsave_for_backward) ctxr rqrrrlrrbroadcast_dimsunsqueeze_scalesunsqueeze_zero_pointstemprymaskr r rforwardos$     "   zFakeQuantPerChannel.forwardcCs|j\}||dddddfSr<) saved_tensors)rgyrr r rbackwardszFakeQuantPerChannel.backwardN)__name__ __module__ __qualname__ staticmethodrrr r r rrns  rfake_quant_per_channelAutogradcCst||||||Sr<)rapplyr rqrrrlrrr r rrs cCs t|Sr<r r/rr r rfake_quant_per_channel_metarrzFconvert_element_type.no_fuse(Tensor input, ScalarType dtype) -> Tensorzconvert_element_type.no_fusecCstjjj||Sr<)r opsprimsconvert_element_typedefaultrr r rrsrcCstj||dSrDrrr r rconvert_element_type_metasr)r)Irtypingrr torch._refsrtorch.ao.quantization.utilsrr torch.libraryrrquantized_decomposed_libuint8ruint16int16r_INTEGER_DTYPES float8_e5m2 float8_e4m3fn _FLOAT_DTYPESrupdaterdefinerVrrrrr1r9r:r;r=r>rErHrIrJrKrjrZr]rdrerorprzr{r~rrrrrrrrr)rrrrrautogradFunctionrrrrrr r r rs       $        2            ' '     /     >    #   +      &    !   $ %  .     "