3jy@X SSKrSSKJr SSKJr SSKJr SSKJ r SSK J r J r J r Jr SSKJrJrJr SS KJr S r/S Qr\R,4S \R.S \R.S\R0S\R.4Sjjr\R44S \R.S \R.S\R0S\R.4Sjjr\R,4S \R.S\R.S\R0S\R.4SjjrSJS \R.S\R.S\S\S\S\\R.\R.44 Sjjr \R4S4S\R.S\R.S\R.S\R0S\S\R.4 Sjjr!SKS\R.S\R.S\R.S \R.S!\R.S"\R.S#\R.S-S$\R0S-S%\R.S-S\R.4S&jjr"SLS \R.S'\S\\R.\R.44S(jjr#\R44S\R.S\R.S\R0S\R.4S)jjr$SMS\R.S\R.S!\R.S"\R.S#\R.S-S$\R0S-S\R.4S*jjr%SNS \R.S+\R.S,\R.S-\&S.\S/\R.S-S\\R.\R.\R.44S0jjr'SOS1\R.S2\R.S3\R.S4\R.S5\R.S6\R.S#\R.S-S.\S\R.4S7jjr(SPS \R.S8\R.S4\R.S9\R.S#\R.S-S:\&S\R.4S;jjr)S<\R.S=\R.S>\R.S\\R.\R.44S?jr*S \R.S>\R.S\R.4S@jr+S<\R.S=\R.S>\R.S\\R.\R.44SAjr,S \R.S>\R.S\R.4SBjr-SC\.\/SS4SDjr0SE\/SS4SFjr1SE\/SS4SGjr2S\34SHjr4SE\/4SIjr5g)QN)cuda)eager)triton) DTYPE_TO_CODE) BackendErrorBackendNotFoundErrorBackendNotImplementedErrorNoCapableBackendError) from_blocked swap_nibbles to_blocked)registryz0.1.0)quantize_per_tensor_fp8dequantize_per_tensor_fp8quantize_nvfp4dequantize_nvfp4quantize_mxfp8dequantize_mxfp8quantize_svdquant_w4a4scaled_mm_nvfp4scaled_mm_mxfp8scaled_mm_svdquant_w4a4gemv_awq_w4a16 apply_rope apply_rope1apply_rope_split_halfapply_rope_split_half1r rr list_backendsset_backend_priorityenable_backenddisable_backendstochastic_rounding_fp8 use_backendrr r r xscale output_typereturnch[Un[RRR XU5$)zQuantize tensor to FP8 format with per-tensor scaling. Args: x: Input tensor scale: Scale tensor (scalar) output_type: FP8 dtype (float8_e4m3fn or float8_e5m2) Returns: Quantized FP8 tensor )rtorchops comfy_kitchen quantize_fp8r%r&r' dtype_codes P/home/wildlama/miniconda3/lib/python3.13/site-packages/comfy_kitchen/__init__.pyrrAs+{+J 99 " " / /* EEch[Un[RRR XU5$)aDequantize tensor from FP8 format with per-tensor scaling. Args: x: Input FP8 tensor (float8_e4m3fn or float8_e5m2) scale: Scale tensor (scalar) output_type: Target dtype (float32, float16, or bfloat16) Returns: Dequantized tensor in specified output format )rr*r+r,dequantize_fp8r.s r0rrTs+{+J 99 " " 1 1!J GGr1rngcFXUS.n[R"SUS9nU"S0UD6$)zStochastically round tensor to FP8 format. Args: x: Input tensor rng: Random uint8 tensor with the same shape as x output_type: FP8 dtype (float8_e4m3fn or float8_e5m2) Returns: Stochastically rounded FP8 tensor )r%r4r'r#)kwargs)rget_implementation)r%r4r'r6impls r0r#r#gs- =F  & &'@ PD >&>r1Tper_tensor_scaleepsilonpad_16xhi_firstcX[RRRXX#U5$)a Quantize tensor to NVFP4 format with block-wise scaling. Args: x: Input tensor (2D) per_tensor_scale: Global scale factor epsilon: Epsilon for numerical stability pad_16x: If True, implicit zero-padding is applied to make dimensions divisible by 16 hi_first: Nibble packing order. If True (default), the even-indexed element is stored in the high nibble of each packed byte. If False, the even-indexed element is stored in the low nibble. Returns: Tuple of (quantized_tensor, block_scales) )r*r+r,r)r%r:r;r<r=s r0rr{s$* 99 " " 1 1!wYa bbr1qx block_scalescj[Un[RRR XX%U5$)aDequantize tensor from NVFP4 format with block-wise scaling. Args: qx: Quantized FP4 tensor (packed as uint8) per_tensor_scale: Global scale factor block_scales: Block scales in swizzled layout (float8_e4m3fn) output_type: Target output dtype (float32, float16, or bfloat16) hi_first: Nibble packing order. Must match the packing order used during quantization. If True (default), the even-indexed element is in the high nibble. Returns: Dequantized tensor in specified output format )rr*r+r,r)r?r:r@r'r=r/s r0rrs.*{+J 99 " " 3 3B,dl mmr1abtensor_scale_atensor_scale_b block_scale_a block_scale_bbias out_dtypealphac Uc[Rn[Un [RRR XX#XEXiU5 $)aBMatrix multiplication with NVFP4 quantized inputs. Computes: y = (a @ b.T) * (tensor_scale_a * tensor_scale_b) + bias Args: a: Quantized matrix A (M, K//2) in uint8 format b: Quantized matrix B (N, K//2) in uint8 format tensor_scale_a: Global scale for A tensor_scale_b: Global scale for B block_scale_a: Block-wise scales for A block_scale_b: Block-wise scales for B bias: Optional bias vector out_dtype: Output dtype (defaults to bfloat16) alpha: Output scale (tensor_scale_a * tensor_scale_b) Returns: Result tensor of shape (M, N) )r*bfloat16rr+r,r) rBrCrDrErFrGrHrIrJr/s r0rrsF:NN y)J 99 " " 2 2 nd r1pad_32xcT[RRRX5$)aQuantize tensor to MXFP8 format with block-wise E8M0 scaling. MXFP8 uses block size 32 with power-of-2 (E8M0) block scales. Args: x: Input tensor (2D, shape M x K, K must be divisible by 32) pad_32x: If True, pad dimensions to be divisible by 32 Returns: Tuple of (quantized_fp8_tensor, block_scales_e8m0) - quantized_fp8_tensor: FP8 E4M3 data of shape (M, K) - block_scales_e8m0: E8M0 scales in swizzled layout )r*r+r,r)r%rMs r0rrs" 99 " " 1 1! ==r1ch[Un[RRR XU5$)a"Dequantize tensor from MXFP8 format. Args: qx: Quantized FP8 tensor (float8_e4m3fn) block_scales: E8M0 block scales in swizzled layout (float8_e8m0fnu) output_type: Target output dtype (float32, float16, or bfloat16) Returns: Dequantized tensor in specified output format )rr*r+r,r)r?r@r'r/s r0rrs+{+J 99 " " 3 3Bj QQr1cUc[Rn[Un[RRR XX#XF5$)aMatrix multiplication with MXFP8 quantized inputs. Computes: y = a @ b.T + bias Args: a: Quantized FP8 matrix A (M, K) b: Quantized FP8 matrix B (N, K) block_scale_a: E8M0 block scales for A in swizzled layout block_scale_b: E8M0 block scales for B in swizzled layout bias: Optional bias vector out_dtype: Output dtype (defaults to bfloat16) Returns: Result tensor of shape (M, N) )r*rLrr+r,r)rBrCrFrGrHrIr/s r0rrsA.NN y)J 99 " " 2 2 mD r1smooth lora_downpad_size act_unsignedlora_xcX[RRRXX#XE5$)uQuantize activations to int4 with smoothing + LoRA down projection. Args: x: (M, K) bf16/fp16 main-path input (caller pre-shifts if unsigned path). smooth: (K,) smoothing factor applied before quantization. lora_down: (K, R) low-rank down projection weight. pad_size: pad M to multiple of this value (default 256). act_unsigned: if True, quantize into uint4 [0, 15] (scale=max/15) for u4 MMA downstream. Caller must ensure x is non-negative — the shift constant is a model-topology concern, not part of this op. lora_x: (M, K) optional pre-shift activation for LoRA. Defaults to x. Pass raw (un-shifted) x when x has been pre-shifted for unsigned path. Returns: (quantized_x uint8 [M_pad, K//2], ascales [K//64, M_pad], lora_act [M_pad, R]) Note: eager returns fp32 lora_act as a high-precision reference. The CUDA backend returns lora_act in x.dtype because the runtime epilogue consumes it as bf16/fp16; this avoids an otherwise redundant cast/allocation. )r*r+r,r)r%rQrRrSrTrUs r0rrs(8 99 " " 9 9 9  r1actwgtascaleswscales lora_act_inlora_upc Z[RRRXX#XEXg5$)aFSVDQuant W4A4 int4 GEMM + LoRA-up + bias. Computes out = int4_matmul(act, wgt, ascales, wscales) + lora_act_in @ lora_up^T + bias. The CUDA backend performs int4 MMA + per-group dequant + bias in one kernel and, when lora_act_in/proj_up layout and dtype allow it, fuses LoRA-up into the same writeback epilogue with bf16/fp16 tensor-core MMA. Unsupported combinations fall back to the wrapper's bf16/fp16 addmm_ path. Args: act: (M, K//2) uint8 packed activations from quantize_svdquant_w4a4. wgt: (N, K//2) int8 packed weights (natural row-major), or backend specific tile-packed storage. ascales: (K//64, M) activation scales. wscales: (K//64, N) weight scales. lora_act_in: (M, R) LoRA activations from quantize step. lora_up: (N, R) LoRA up projection weight, or matching tile-packed storage for tile-packed weights. bias: optional (N,) bias. act_unsigned: if True, activations are interpreted as unsigned [0,15] by u4.s4 MMA (for post-GELU+shift fc2). Caller pre-shifts. Returns: (M, N) output tensor (same dtype as lora_up). )r*r+r,r)rWrXrYrZr[r\rHrTs r0rr9s+D 99 " " : : 'K$ r1qweightwzeros group_sizecX[RRRXX#XE5$)a|AWQ W4A16 quantized GEMV (for modulation-style layers called with small batch). Args: x: (..., K) bf16/fp16 input. qweight: (N//4, K//2) int32 packed weight. wscales: (K//group_size, N) per-group scales. wzeros: (K//group_size, N) per-group zero points. bias: optional (N,) bias. group_size: quantization group size. Returns: (..., N) output tensor. )r*r+r,r)r%r^rZr_rHr`s r0rr`s(* 99 " " 1 1 GT r1xqxk freqs_ciscV[RRRXU5$)aApply Rotary Position Embedding (RoPE) to query and key tensors. Interleaved layout: pair k uses adjacent elements [2k, 2k+1]. Args: xq: Query tensor xk: Key tensor freqs_cis: Precomputed frequency tensor Returns: Tuple of (transformed_query, transformed_key) )r*r+r,rrbrcrds r0rrzs!" 99 " " - -bi @@r1cT[RRRX5$)zApply Rotary Position Embedding (RoPE) to a single tensor. Interleaved layout: pair k uses adjacent elements [2k, 2k+1]. Args: x: Input tensor freqs_cis: Precomputed frequency tensor Returns: Transformed tensor )r*r+r,rr%rds r0rrs 99 " " . .q <rs8 +./6 @? # \ %22F ||F <<FF \\ F, %~~H ||H <<HH \\ H, %22 ||  \\ . c ||cllcc c  c  5<< %& c8 %~~ n nlln,,n n  n  \\ n@!%$(!%# ||# ||#LL#LL # << # << # ,, #{{T!# <<$ # \\#P> ||> > 5<< %&>. %~~R R,,RR \\ R0!%$(  || ||<<<<  ,,   {{T!  \\F"&  || LL||    LL4   5<<u|| 34P!%$ $ $\\$\\ $  $ \\ $ ,, $$ \\$X!%  || \\\\ LL  ,,     \\4A A A||A 5<< %& A(= ||=||= \\=$L L L||L 5<< %& L0H ||H||H \\H6$49$$#$$t$" &c &r1