3jIb SSKrSSKJrJrJrJrJrJrJrJ r J r J r J r SSK JrJrJr \R"\R$\R&\R(\R*S.r\R/5VVs0sHupX_M snnr\R(4S\R2S\R2S\R4S\R24S jjr\R&4S\R2S\R2S\R4S\R24S jjrS r\R(4S\R2S \R2S\R4S\R24S jjrSCS\R2S\R2S\S\ S\ S\!\R2\R244 Sjjr"\RF"/SQ5RIS5r%0r&\R&S4S\R2S\R2S\R2S\R4S\ S\R24 Sjjr'SDS\R2S\R2S\R2S\R2S\R2S\R2S\R2S-S \R4S-S!\R2S-S\R24S"jjr(S#r)SES\R2S$\ S\!\R2\R244S%jjr*\R&4S\R2S\R2S\R4S\R24S&jjr+SFS\R2S\R2S\R2S\R2S\R2S-S \R4S-S\R24S'jjr,\RZR]S(S)S*9S\R2S\R2S+\/S\R24S,j5r0\0RbS-5r2\RZR]S.S)S*9S\R2S\R2S+\/S\R24S/j5r3\3RbS05r4\RZR]S1S)S*9S\R2S\R2S\S\ S\ S\!\R2\R244 S2j5r5\5RbS35r6\RZR]S4S)S*9S\R2S\R2S\R2S+\/S\ S\R24 S5j5r7\7RbS65r8\RZR]S7S)S*9S\R2S\R2S\R2S\R2S\R2S\R2S\R2S-S+\/S!\R2S-S\R24S8j5r9\9RbS95r:\RZR]S:S)S*9S\R2S$\ S\!\R2\R244S;j5r;\;RbS<5r<\RZR]S=S)S*9S\R2S\R2S+\/S\R24S>j5r=\=RbS?5r>\RZR]S@S)S*9S\R2S\R2S\R2S\R2S\R2S-S+\/S\R24SAj5r?\?RbSB5r@gs snnf)GN) E8M0_BIAS F4_E2M1_MAX F8_E4M3_MAX F8_E5M2_MAX_f32_to_floatx_unpacked _float8_round e8m0_to_f32 from_blocked pack_uint4roundup to_blocked) ScalingType SwizzleType scaled_mm_v2)rxscale output_typereturncU[R:Xa[nO*U[R:Xa[nO[ SUS35eUSU- R UR5-n[R"XC*X4S9nUR U5$)NUnsupported output_type: 3. Expected torch.float8_e4m3fn or torch.float8_e5m2?)out) torch float8_e4m3fnr float8_e5m2r ValueErrortodtypeclamp)rrrlp_maxtemps c/home/wildlama/miniconda3/lib/python3.13/site-packages/comfy_kitchen/backends/eager/quantization.pyquantize_per_tensor_fp8r()se))) )) )' }4g h   e ( (D ;;tWf 7D 77; cDURUS9URUS9-nU$Nr#)r")rrr dq_tensors r'dequantize_per_tensor_fp8r.9s*;'%(((*EEI r)c[R"UUSX- -- S- SU--USU*S-U- -- 5nXeRURS9S-- nUR 5SU-- $)N@rrrr,gp?)rwherer"r#floor)abs_xexponent normal_mask MANTISSA_BITS EXPONENT_BIASrngmantissa_scaleds r' calc_mantissar:@skk #(23 4s :q-?OP #=.1,}<= >O vvO$9$9v:kJJO  "a&6 77r)r8cU[R:XaSup4nO)U[R:XaSup4nO[SUS35eUR 5n[R "U5nUR 5n[R"US:HSU5n[R"[R"[R"U55U-SSU-S- 5nUS:H)n [XxXXQ5USS&U[R"U SX- -S U--SU*S--U-5-n[R"U5n [R"XjRU RUS 9 URU5$) N))rrrrrrrr0r)minmaxr)rrr r!halfsignabsr1r$r2log2r:finfor@rAr") rr8r EXPONENT_BITSr6r7rCr3r4r5infos r'stochastic_rounding_fp8rIKs\ e)))6=3 m )) )6>3 m' }5# #  A ::a=D EEGE ;;uz1d +D{{ EJJu%&6  =1H M"KUk-]E!HEKK ) *sU{; -!# $- D ;;{ #D KK((d; 77; r)Tper_tensor_scaleepsilonpad_16xhi_firstc2URnU(anURupg[US5n[US5n X:wdX:wa>[RRR USX- SX- 45nURnSn UR USSU 5n[R"[R"U5SS9n U R[R5[- n X- n [R"U [S9n[U5nX-nUS:Hn[R"U[R "U5U5nUR#5UR%S5- n[R"UR%S5[R&"U5U5nUn[R"U[*[5nUR)U5n[+USS5n[-UUS9n[/UR[R05S S 9nUU4$) Nrdim)rArr)rMFflatten)shaper rnn functionalpadreshapeamaxrDr"float32rr$rrr1 ones_likefloat unsqueeze zeros_likeviewrr r r)rrJrKrLrM orig_shaperowscols padded_rows padded_cols block_sizemax_abs block_scalescaled_block_scalesscaled_block_scales_fp8scaled_block_scales_fp32 total_scalezero_scale_masktotal_scale_safe data_scaled out_scalesdata_lpblocked_scaless r'quantize_nvfp4rsssJWW dB' dB'  +"5##''A{/A1kFX+YZAJJ *Q-Z0Ajj12.G**U]]+k9K%8#kk*=;O,-DE"=K#a'O{{?EOOK4PR]^'').88<F Mr)abtensor_scale_atensor_scale_b block_scale_a block_scale_bbias out_dtypealphac [UR[R5UR[R5R 5URS5U/URS5U/UU[ R [ R/[ R [ R/[R[R/[R[R/S9 n U $)NrPscale_ascale_brrscale_recipe_ascale_recipe_b swizzle_a swizzle_b) rr`rfloat4_e2m1fn_x2tr BlockWise1x16 TensorWiserSWIZZLE_32_4_4 NO_SWIZZLE) rrrrrrrrrrs r'scaled_mm_nvfp4rs u%%& u%%&((*##B'8##B'8 %33[5K5KL%33[5K5KL //1G1GH //1G1GH F Mr) pad_32xcURnUR5S:XdS5eU(aoURup4[US5n[US5nXS:wdXd:wa>[RR R USXd- SXS- 45nURnO(URS[-S:XdS[35eUup4U[-nURX7[5n[R"[R"U5SS9n U R5[- n [R"U S S 9n [R"U 5n [R"U 5R![R"5[$-n [R"U SS 5n U R![R&5n [)U 5nU S:Hn[R*"U[R,"U5U5nUR5UR/S5- n[R*"UR/S5[R0"U5U5n[R"U[*[5nURU5R![R25n[R*"U[R0"U 5U 5n [5U S S 9nUUR7[R854$)aQuantize tensor to MXFP8 format with block-wise E8M0 scaling. MXFP8 uses block size 32 with power-of-2 (E8M0) block scales. Args: x: Input tensor (2D, shape M x K) output_type: FP8 output dtype (float8_e4m3fn) pad_32x: If True, pad dimensions to be divisible by 32 Returns: Tuple of (quantized_fp8_tensor, block_scales_e8m0) - quantized_fp8_tensor: FP8 data of shape (M, K) or padded shape - block_scales_e8m0: E8M0 scales of shape (M, K//32) in swizzled layout rzInput must be 2Drrrz!K dimension must be divisible by rPrQg8)r@FrS)rUrRr rrVrWrXMXFP8_BLOCK_SIZErYrZrDr]rr$rEceilr"int32ruint8r r1r\r^r_rr r`float8_e8m0fnu)rrrarbrcrdre num_blocks x_blockedrg scale_needed log2_scale exp_biasedblock_scales_e8m0block_scales_f32 zero_maskrodata_fp8rrs r'quantize_mxfp8rsl$J 557a<+++<WW dB' dB'  +"5##''A{/A1kFX+YZAJwwqz,,1i5VWgVh3ii1JD))J $,<=Ijj9-26G==?[0L;;|;LL)JJ'**5;;7)CJZC0J" ekk2"#45AI{{9eoo>N.OQab//#&6&@&@&DDK++i11"5u7G7G 7TVabK++kK<EK"":.11%2E2EFH Iu/?/?@Q/RTef 15AN ^(()=)=> >>r)cjURnUupEU[-nUR[R5n[ UUUS9n[ U5n UR[R5RXF[5n XRS5-n U RU5RU5$)zDequantize tensor from MXFP8 format. Args: qx: Quantized FP8 tensor (float8_e4m3fn or float8_e5m2) block_scales: E8M0 block scales in swizzled layout (float8_e8m0fnu) output_type: Target output dtype Returns: Dequantized tensor rxrP) rUrr`rrr r r"r[rYr^) rurvrrarbrcrblock_scales_uint8rrdata_f32rs r'dequantize_mxfp8rGsJJD))J&**5;;7*##:;uuU]]#++D>NOH "<" 0&'89L\ BF  & &'@ & PD >&>r)c<[Un[R"XS9$r+rr empty_likerrrrs r'_op_quantize_fp8_faker&'89L   A 22r)zcomfy_kitchen::dequantize_fp8cZSSKJn [UnXUS.nURSUS9nU"S0UD6$)a"Dequantize tensor from FP8 format with per-tensor scaling. Args: x: Input FP8 tensor (float8_e4m3fn or float8_e5m2) scale: Scale tensor (scalar) output_dtype_code: Target dtype code (0=float32, 1=float16, 2=bfloat16) Returns: Dequantized tensor in specified output format rrrr.rrrrs r'_op_dequantize_fp8rs> 0&'89L\ BF  & &'B6 & RD >&>r)c<[Un[R"XS9$r+rrs r'_op_dequantize_fp8_fakerrr)zcomfy_kitchen::quantize_nvfp4cJSSKJn XX#US.nURSUS9nU"S0UD6$)aQuantize tensor to NVFP4 format with block-wise scaling. Args: x: Input tensor (2D) per_tensor_scale: Global scale factor epsilon: Epsilon for numerical stability pad_16x: If True, pad dimensions to be divisible by 16 hi_first: If True, the even-indexed element is packed into the high nibble (default). If False, the even-indexed element goes into the low nibble. Returns: Tuple of (quantized_tensor, block_scales) rr)rrJrKrLrMrsrrrrr)rrJrKrLrMrrrs r'_op_quantize_nvfp4rs6*0wpx yF  & &'7 & GD >&>r)cZURupVU(a[US5n[US5n[R"XVS-4[RUR S9n[US5n[US-S5n [R"X4[R UR S9n Xz4$)NrOrr#r}r<)rUr remptyrr}r) rrJrKrLrMrbrcqdata scale_rows scale_colsrvs r'_op_quantize_nvfp4_fakersJDtR tR  KKqy)QXX NEs#JQ'J;; 7u?R?R[\[c[cdL  r)zcomfy_kitchen::dequantize_nvfp4c\SSKJn [UnXX&US.nURSUS9nU"S0UD6$)aDequantize tensor from NVFP4 format with block-wise scaling. Args: qx: Quantized FP4 tensor (packed as uint8) per_tensor_scale: Global scale factor block_scales: Block scales in swizzled layout (float8_e4m3fn) output_dtype_code: Target dtype code (0=float32, 1=float16, 2=bfloat16) hi_first: If True, the high nibble is the even-indexed element (default). If False, the low nibble is the even-indexed element. Returns: Dequantized tensor in specified output format rr)rurJrvrrMrrrr) rurJrvrrMrrrrs r'_op_dequantize_nvfp4rsH*0&'89LlFNOF  & &'9& & ID >&>r)cv[UnURupg[R"XgS-4XPRS9$)NrrrrUrrr})rurJrvrrMrrb cols_packeds r'_op_dequantize_nvfp4_faker#s5&'89LD ;;Ao.l99 UUr)zcomfy_kitchen::scaled_mm_nvfp4c `SSKJn [Un XX#XEXjUS. n U RSU S9n U "S0U D6$)a[Matrix multiplication with NVFP4 quantized inputs. Computes: y = (a @ b.T) * (tensor_scale_a * tensor_scale_b) + bias Args: a: Quantized matrix A (M, K//2) in uint8 format b: Quantized matrix B (N, K//2) in uint8 format tensor_scale_a: Global scale for A tensor_scale_b: Global scale for B block_scale_a: Block-wise scales for A block_scale_b: Block-wise scales for B bias: Optional bias vector output_dtype_code: Output dtype code (0=float32, 1=float16, 2=bfloat16) alpha: Output scale (tensor_scale_a * tensor_scale_b) Returns: Result tensor of shape (M, N) rr) rrrrrrrrrrrrr) rrrrrrrrrrrrrs r'_op_scaled_mm_nvfp4r+sK<0#$56I (&u F  & &'8 & HD >&>r)c [Un URSn URSn [R"X4XRS9$Nrrr) rrrrrrrrrrmns r'_op_scaled_mm_nvfp4_fakerVs@ $$56I  A  A ;;vYxx @@r)zcomfy_kitchen::quantize_mxfp8cFSSKJn XS.nURSUS9nU"S0UD6$)a&Quantize tensor to MXFP8 format with block-wise E8M0 scaling. MXFP8 uses block size 32 with power-of-2 (E8M0) block scales. Args: x: Input tensor (2D, shape M x K) pad_32x: If True, pad dimensions to be divisible by 32 Returns: Tuple of (quantized_fp8_tensor, block_scales_e8m0) rr)rrrrrr)rrrrrs r'_op_quantize_mxfp8res1 0 )F  & &'7 & GD >&>r)cXURup#U(a[US5n[US5n[R"X#4[RUR S9nUS-n[US5n[US5n[R"Xg4[R UR S9nXH4$)Nrrrr<)rUr rrrr}r) rrrbrcrrrrrvs r'_op_quantize_mxfp8_faker|sJDtR tR  KK E,?,? QEJs#JQ'J;; 7u?S?S\]\d\deL  r)zcomfy_kitchen::dequantize_mxfp8cZSSKJn [UnXUS.nURSUS9nU"S0UD6$)a)Dequantize tensor from MXFP8 format. Args: qx: Quantized FP8 tensor (float8_e4m3fn) block_scales: E8M0 block scales in swizzled layout (float8_e8m0fnu) output_dtype_code: Target dtype code (0=float32, 1=float16, 2=bfloat16) Returns: Dequantized tensor in specified output format rr)rurvrrrrr)rurvrrrrrs r'_op_dequantize_mxfp8rs> 0&'89L\ RF  & &'9& & ID >&>r)c<[Un[R"XS9$r+r)rurvrrs r'_op_dequantize_mxfp8_fakers&'89L   B 33r)zcomfy_kitchen::scaled_mm_mxfp8c\SSKJn [UnXX#XGS.nURSUS9n U "S0UD6$)aMatrix multiplication with MXFP8 quantized inputs. Computes: y = a @ b.T + bias Args: a: Quantized FP8 matrix A (M, K) b: Quantized FP8 matrix B (N, K) block_scale_a: E8M0 block scales for A in swizzled layout block_scale_b: E8M0 block scales for B in swizzled layout bias: Optional bias vector output_dtype_code: Output dtype code (0=float32, 1=float16, 2=bfloat16) Returns: Result tensor of shape (M, N) rr)rrrrrrrrrr) rrrrrrrrrrs r'_op_scaled_mm_mxfp8rsF00#$56I &F  & &'8 & HD >&>r)c[UnURSnURSn[R"Xx4X`RS9$rr) rrrrrrrrrs r'_op_scaled_mm_mxfp8_fakers@$$56I  A  A ;;vYxx @@r))rtFT)NNN)F)NN)Arcomfy_kitchen.float_utilsrrrrrrr r r r r comfy_kitchen.scaled_mm_v2rrrr[float16bfloat16rr ritems DTYPE_TO_CODETensorr#r(r.r:rIr]booltuplerstensorr^r~r{rrrrrrlibrary custom_oprr register_fakerrrrrrrrrrrrrrr)kvs00r'r sV     NM }} }} ~~   #6";";"=>"=$!"=> FKEXEX  || !LL 7<{{  \\ "FK^^ ||!LL7<{{ \\8 %22%  ||% % %  \\ % V -# ||-#ll-#-# -#  -#  5<< %& -#` <<  9Q<  %~~ 0 0ll0,,0 0  0  \\ 0t!%$(!% || ||LLLL  <<  <<  ,, {{T! <<$  \\@ C? ||C? C? 5<< %&C?R %~~%@ %@,,%@%@ \\ %@Z!%$( # ||# ||#<<#<< # ,,  # {{T! # \\#X6RH || << \\ I.3 3 8rJ || << \\ K.!!3"3 8rJ ||ll     5<< %& K6!!"$:L ll,,     \\ M8##V$V9K' ||' ||'LL'LL ' << ' << ' ,, '' <<$ ' \\'L'T""A#A8rJ ||  5<< %&K,!!"(:L ,, \\ M.##4$4 9K  ||  || << <<  ,,     \\ L F""A#A[?s6[