3j7SSKJr SSKJr SSKJr SSKJrJrJ r SSK J r J r J r JrJr \ "5(a SSKrSSKJr \ "5(aSS KJr \"5r\ "5(a \(dSSKr\R."\5r"S S \5r"S S \R"R65r"SS\R:5r\"SS9S5rSS\ \!S-4Sjjr"g)) lru_cache)ACT2FN) ConversionOps)get_module_from_name on_deviceshould_convert_module)is_accelerate_availableis_fbgemm_gpu_availableis_torch_availableis_torch_xpu_availableloggingN)nn)init_empty_weightsc \rSrSrSrS S\\\R\ \R-4S\RRS-S\\\R44Sjjr Sr g) FbgemmFp8Quantize,cXlgN hf_quantizer)selfrs ^/home/wildlama/miniconda3/lib/python3.13/site-packages/transformers/integrations/fbgemm_fp8.py__init__FbgemmFp8Quantize.__init__-s(N input_dictmodelreturnc Z[UR55SupEUSnSSKJn [ X$5upx[ Xv5(Ga.US:XaUR SS5n U Rn U RSU S5n [U 5 [U 5upSSS5 W RU 5nUR SS5nW RU SSU S5nOUS:XaUR SS5n U Rn U RSU S5n [U 5 [U 5upSSS5 W RU 5nUR SS5nW RU SU SS5nO][U5 [U5upSSS5 [RRWRURSS55nU[RRW5US3W0$!,(df  GNf=f!,(df  N=f!,(df  N=f) Nrr)FbgemmFp8Llama4TextExperts gate_up_proj down_proj_scale)tupleitems integrationsr!r isinstance transposeshapereshaperquantize_fp8_per_rowtorchr Parameterview)rrrkwargs target_keyvaluer!module tensor_nametransposed_paramoriginal_shapeflattened_paramnew_value_flatweight_scale_flat new_value weight_scales rconvertFbgemmFp8Quantize.convert0s  "*"2"2"45a8 a=25E f 9 9n,$)??1a#8 "2!7!7"2":":2~b?Q"R/8L_8]5N0+22>B %//15 0889JA~^_O`a  +$)??1a#8 "2!7!7"2":":2~b?Q"R/8L_8]5N0+22>B %//15 0889JN[\L]_`a 5!*>u*E' " 88--l.?.? @R@RST@UWX.YZLEHH..y9j\;PR^__;0/"0/"!s$ G9#H H9 H H H*rr)__name__ __module__ __qualname____firstlineno__rdictstrr/TensorlistrModuler>__static_attributes__rrrr,sn) )-5`ellT%,,-???@5`xx%5` c5<< 5`5`rrcH^\rSrSr\R 4U4SjjrSrSrU=r $)FbgemmFp8Linearhc^>[TU]XU5 XlX l[R R [R"X!4US95Ul[R R [R"US4[RS95Ul URS[R"S/[RS9SS9 U(aP[R R [R"UR[RS95Ul gSUl g)Ndtyper#input_scale_ubF persistent)superr in_features out_featuresr/rr0zerosweightfloat32r=register_bufferfloatbias)rrUrVr\rP __class__s rrFbgemmFp8Linear.__init__is D9&(hh((l5PX])^_ !HH..u{{L!;LTYTaTa/bc -u{{A3ekk/R_de **5;;8I8IRWR_R_+`aDIDIrc /URSSQSP7n[U5 [URSURS5R 5UR S9up4SSS5 UR R[R5n[(ac[R"WURR5WRS5UR5URUR S9nOR[R"R$R'WURWUSS9nUR bX`R -OUnUR)U5nAAU$!,(df  GN =f)Nr$)scale_ub)scale_ascale_b out_dtyper\Tuse_fast_accum)r,rr.r1 contiguousrQr=tor/rY_is_torch_xpu_available _scaled_mmrXt unsqueezerPr\opsfbgemmf8f8bf16_rowwiser-)rx output_shape x_quantizedx_scaleweight_scale_float32outputs rforwardFbgemmFp8Linear.forwardws3*"*r* q\#7r1772;'224t?R?R$ K $0033EMMB " "%% ))"-,..0''YY FYY%%66T[['3GX\7F,099+@Vii'fF -  -\s AE(( E7)r\rUrVrXr=) r@rArBrCr/ float8_e4m3fnrrurI __classcell__r]s@rrLrLhs>C>Q>Q rrLcH^\rSrSr\R 4U4SjjrSrSrU=r $)r!cT>[TU]5 URUlURUlUR UlURUl[URUl [RR[R"URUR SUR -4[RS95Ul[RR[R"URSUR S-4[R S95Ul[RR[R"URUR UR 4[RS95Ul[RR[R"URUR S4[R S95UlUR)S[R"S/[R*S9SS9 g)NrrOr#rQFrR)rTrnum_local_experts num_expertsintermediate_size hidden_size expert_dimr hidden_actact_fnr/rr0rWrwr"rYgate_up_proj_scaler%down_proj_scalerZr[)rconfigrPr]s rr#FbgemmFp8Llama4TextExperts.__init__s !33!'!9!9!--00V../ !HH.. KK))4+;+;Q=PQY^YlYl m #((("4"4 KK))1doo.AB%-- X# ++ KK))4??DPQVVWY[\]hhjlln+11 %%$%%a(221a89K9LMXXZ\\^(222669!<=O=PQVVWY[\]hhjlln+11 yy''88$%%a(221a89L:LMXXZ .1!45H6HINNrSTU``b#' 9YY%%66$%%a(221a89K9LMXXZ .1!45G5HINNrSTU``b#' 7[[..I9%7KIcgcvcv7w4#&'+&:&:&=&=emm&L #&& % 0 0'NN1%//15@@B+55b93A6;;BBMMOQQS+11 ! !& 0 0 A A'NN1%//15@@B#+A.33B:EEG#' !B! +KNE)F"nn]%9%9: D$4$45532P&%sS-S?- S< ? T ) rr%rrr"rrrr~) r@rArBrCr/rYrrurIrxrys@rr!r!s%*]]f0R6R6rr!r#)maxsizec[(aSSKJn U"S5R$[R R R$)Nr# get_kernelzkernels-community/fp8-fbgemm)rh hub_kernelsrr.r/rlrmrs rget_quantize_fp8_per_rowrs3+89NNN 99   0 00rmodules_to_not_convertc[5qSnU(a0OSS0nUR5Hupx[Xq5(dMSn [ SS9 UR R S:Xa@[URSUR5n [U =(d UR5n O^[U[R5(a?[URURUR SL40UD6n U R#S5 SSS5 U cMUR%Xy5 SnM U(d[&R)S5 U$!,(df  NH=f) a A helper function to replace all `torch.nn.Linear` modules by `FbgemmFp8Linear` modules. This will enable running your models using high performance fp8 kernel from FBGEMM library. Parameters: model (`torch.nn.Module`): Input model or `torch.nn.Module` as the function is run recursively. modules_to_not_convert (`list[`str`]`, *optional*, defaults to `None`): Names of the modules to not convert. In practice we keep the `lm_head` in full precision for numerical stability reasons. quantization_config (`FbgemmFp8Config`): The quantization config object that contains the quantization parameters. pre_quantized (`book`, defaults to `False`): Whether the model is pre-quantized or not FrPNT)include_buffersLlama4TextExperts text_configzYou are loading your model using FP8 quantization but no linear modules were found in your model. Please double check your model architecture, or submit an issue on github if you think this is a bug.)rr. named_modulesr rr]r@getattrrr!r*rLinearrLrUrVr\requires_grad_ set_submoduleloggerwarning) rrquantization_config pre_quantizedtp_planhas_been_replaced module_kwargs module_namer5 new_modulers rreplace_with_fbgemm_fp8_linearr s.$45'Bgt_M$224 $[II   5((,?? &ellM5<<P 7 8Su||T FBII..,&&''KKt+$  ))%06"    K4 5 58   L=6 5s B9E E )NNFN)# functoolsr activationsrcore_model_loadingrquantizers.quantizers_utilsrrr utilsr r r r rr/r acceleraterrhfbgemm_gpu.experimental.gen_ai fbgemm_gpu get_loggerr@rrrrLrHr!rrGrErrJrrrs .``-02%<)   H %9` 9`x+ehhoo+\k6k6\ 111tx:#'9t#3:r