+jy@ddlZddlmZddlmZddlmZddlm Z ddl m Z m Z m Z mZddlmZmZmZdd lmZd Zgd Zejfd ejd ejdejdejfdZejfd ejd ejdejdejfdZejfd ejdejdejdejfdZ dNd ejdejdedededeejejff dZ ejdfdejdejdejdejdedejf dZ! dOdejd ejd!ejd"ejd#ejd$ejd%ejdzd&ejdzd'ejdzdejfd(Z" dPd ejd)edeejejffd*Z#ejfdejdejdejdejfd+Z$ dQdejd ejd#ejd$ejd%ejdzd&ejdzdejfd,Z% dRd ejd.ejd/ejd0e&d1ed2ejdzdeejejejffd3Z' dSd4ejd5ejd6ejd7ejd8ejd9ejd%ejdzd1edejfd:Z( dTd ejde&dejfd?Z)d@ejdAejdBejdeejejffdCZ*d ejdBejdejfdDZ+d@ejdAejdBejdeejejffdEZ,d ejdBejdejfdFZ-dGe.e/ddfdHZ0dIe/ddfdJZ1dIe/ddfdKZ2de3fdLZ4dIe/fdMZ5dS)UN)cuda)eager)triton) DTYPE_TO_CODE) BackendErrorBackendNotFoundErrorBackendNotImplementedErrorNoCapableBackendError) from_blocked swap_nibbles to_blocked)registryz0.1.0)quantize_per_tensor_fp8dequantize_per_tensor_fp8quantize_nvfp4dequantize_nvfp4quantize_mxfp8dequantize_mxfp8quantize_svdquant_w4a4scaled_mm_nvfp4scaled_mm_mxfp8scaled_mm_svdquant_w4a4gemv_awq_w4a16 apply_rope apply_rope1apply_rope_split_halfapply_rope_split_half1r rr list_backendsset_backend_priorityenable_backenddisable_backendstochastic_rounding_fp8 use_backendrr r r xscale output_typereturncht|}tjj|||S)zQuantize tensor to FP8 format with per-tensor scaling. Args: x: Input tensor scale: Scale tensor (scalar) output_type: FP8 dtype (float8_e4m3fn or float8_e5m2) Returns: Quantized FP8 tensor )rtorchops comfy_kitchen quantize_fp8r%r&r' dtype_codes Y/home/wildlama/comfy/ComfyUI/.venv/lib/python3.11/site-packages/comfy_kitchen/__init__.pyrrAs+{+J 9 " / /5* E EEcht|}tjj|||S)a0Dequantize tensor from FP8 format with per-tensor scaling. Args: x: Input FP8 tensor (float8_e4m3fn or float8_e5m2) scale: Scale tensor (scalar) output_type: Target dtype (float32, float16, or bfloat16) Returns: Dequantized tensor in specified output format )rr*r+r,dequantize_fp8r.s r0rrTs+{+J 9 " 1 1!UJ G GGr1rngcJ|||d}tjd|}|di|S)aStochastically round tensor to FP8 format. Args: x: Input tensor rng: Random uint8 tensor with the same shape as x output_type: FP8 dtype (float8_e4m3fn or float8_e5m2) Returns: Stochastically rounded FP8 tensor )r%r4r'r#)kwargs)rget_implementation)r%r4r'r6impls r0r#r#gs;S = =F  &'@ P P PD 4>>&>>r1FTper_tensor_scaleepsilonpad_16xhi_firstcRtjj|||||S)aLQuantize tensor to NVFP4 format with block-wise scaling. Args: x: Input tensor (2D) per_tensor_scale: Global scale factor epsilon: Epsilon for numerical stability pad_16x: If True, implicit zero-padding is applied to make dimensions divisible by 16 hi_first: Nibble packing order. If True (default), the even-indexed element is stored in the high nibble of each packed byte. If False, the even-indexed element is stored in the low nibble. Returns: Tuple of (quantized_tensor, block_scales) )r*r+r,r)r%r;r<r=r>s r0rr{s(* 9 " 1 1!5EwPWYa b bbr1qx block_scalesclt|}tjj|||||S)aFDequantize tensor from NVFP4 format with block-wise scaling. Args: qx: Quantized FP4 tensor (packed as uint8) per_tensor_scale: Global scale factor block_scales: Block scales in swizzled layout (float8_e4m3fn) output_type: Target output dtype (float32, float16, or bfloat16) hi_first: Nibble packing order. Must match the packing order used during quantization. If True (default), the even-indexed element is in the high nibble. Returns: Dequantized tensor in specified output format )rr*r+r,r)r@r;rAr'r>r/s r0rrs2*{+J 9 " 3 3B8H,Xbdl m mmr1abtensor_scale_atensor_scale_b block_scale_a block_scale_bbias out_dtypealphac | tj}t|} tjj|||||||| | S)azMatrix multiplication with NVFP4 quantized inputs. Computes: y = (a @ b.T) * (tensor_scale_a * tensor_scale_b) + bias Args: a: Quantized matrix A (M, K//2) in uint8 format b: Quantized matrix B (N, K//2) in uint8 format tensor_scale_a: Global scale for A tensor_scale_b: Global scale for B block_scale_a: Block-wise scales for A block_scale_b: Block-wise scales for B bias: Optional bias vector out_dtype: Output dtype (defaults to bfloat16) alpha: Output scale (tensor_scale_a * tensor_scale_b) Returns: Result tensor of shape (M, N) )r*bfloat16rr+r,r) rCrDrErFrGrHrIrJrKr/s r0rrsL:N y)J 9 " 2 2 1nn}dJ  r1pad_32xcLtjj||S)aQuantize tensor to MXFP8 format with block-wise E8M0 scaling. MXFP8 uses block size 32 with power-of-2 (E8M0) block scales. Args: x: Input tensor (2D, shape M x K, K must be divisible by 32) pad_32x: If True, pad dimensions to be divisible by 32 Returns: Tuple of (quantized_fp8_tensor, block_scales_e8m0) - quantized_fp8_tensor: FP8 E4M3 data of shape (M, K) - block_scales_e8m0: E8M0 scales in swizzled layout )r*r+r,r)r%rNs r0rrs" 9 " 1 1!W = ==r1cht|}tjj|||S)a>Dequantize tensor from MXFP8 format. Args: qx: Quantized FP8 tensor (float8_e4m3fn) block_scales: E8M0 block scales in swizzled layout (float8_e8m0fnu) output_type: Target output dtype (float32, float16, or bfloat16) Returns: Dequantized tensor in specified output format )rr*r+r,r)r@rAr'r/s r0rrs+{+J 9 " 3 3B j Q QQr1c| tj}t|}tjj||||||S)aMatrix multiplication with MXFP8 quantized inputs. Computes: y = a @ b.T + bias Args: a: Quantized FP8 matrix A (M, K) b: Quantized FP8 matrix B (N, K) block_scale_a: E8M0 block scales for A in swizzled layout block_scale_b: E8M0 block scales for B in swizzled layout bias: Optional bias vector out_dtype: Output dtype (defaults to bfloat16) Returns: Result tensor of shape (M, N) )r*rMrr+r,r)rCrDrGrHrIrJr/s r0rrsE.N y)J 9 " 2 2 1m]D*  r1smooth lora_downpad_size act_unsignedlora_xcTtjj||||||S)u7Quantize activations to int4 with smoothing + LoRA down projection. Args: x: (M, K) bf16/fp16 main-path input (caller pre-shifts if unsigned path). smooth: (K,) smoothing factor applied before quantization. lora_down: (K, R) low-rank down projection weight. pad_size: pad M to multiple of this value (default 256). act_unsigned: if True, quantize into uint4 [0, 15] (scale=max/15) for u4 MMA downstream. Caller must ensure x is non-negative — the shift constant is a model-topology concern, not part of this op. lora_x: (M, K) optional pre-shift activation for LoRA. Defaults to x. Pass raw (un-shifted) x when x has been pre-shifted for unsigned path. Returns: (quantized_x uint8 [M_pad, K//2], ascales [K//64, M_pad], lora_act [M_pad, R]) Note: eager returns fp32 lora_act as a high-precision reference. The CUDA backend returns lora_act in x.dtype because the runtime epilogue consumes it as bf16/fp16; this avoids an otherwise redundant cast/allocation. )r*r+r,r)r%rSrTrUrVrWs r0rrs.8 9 " 9 9 69h f  r1actwgtascaleswscales lora_act_inlora_upc Xtjj||||||||S)aSVDQuant W4A4 int4 GEMM + LoRA-up + bias. Computes out = int4_matmul(act, wgt, ascales, wscales) + lora_act_in @ lora_up^T + bias. The CUDA backend performs int4 MMA + per-group dequant + bias in one kernel and, when lora_act_in/proj_up layout and dtype allow it, fuses LoRA-up into the same writeback epilogue with bf16/fp16 tensor-core MMA. Unsupported combinations fall back to the wrapper's bf16/fp16 addmm_ path. Args: act: (M, K//2) uint8 packed activations from quantize_svdquant_w4a4. wgt: (N, K//2) int8 packed weights (natural row-major), or backend specific tile-packed storage. ascales: (K//64, M) activation scales. wscales: (K//64, N) weight scales. lora_act_in: (M, R) LoRA activations from quantize step. lora_up: (N, R) LoRA up projection weight, or matching tile-packed storage for tile-packed weights. bias: optional (N,) bias. act_unsigned: if True, activations are interpreted as unsigned [0,15] by u4.s4 MMA (for post-GELU+shift fc2). Caller pre-shifts. Returns: (M, N) output tensor (same dtype as lora_up). )r*r+r,r)rYrZr[r\r]r^rIrVs r0rr9s3D 9 " : : S'7K$   r1@qweightwzeros group_sizecTtjj||||||S)aAWQ W4A16 quantized GEMV (for modulation-style layers called with small batch). Args: x: (..., K) bf16/fp16 input. qweight: (N//4, K//2) int32 packed weight. wscales: (K//group_size, N) per-group scales. wzeros: (K//group_size, N) per-group zero points. bias: optional (N,) bias. group_size: quantization group size. Returns: (..., N) output tensor. )r*r+r,r)r%rar\rbrIrcs r0rr`s.* 9 " 1 1 7GVT:  r1xqxk freqs_ciscNtjj|||S)a7Apply Rotary Position Embedding (RoPE) to query and key tensors. Interleaved layout: pair k uses adjacent elements [2k, 2k+1]. Args: xq: Query tensor xk: Key tensor freqs_cis: Precomputed frequency tensor Returns: Tuple of (transformed_query, transformed_key) )r*r+r,rrerfrgs r0rrzs!" 9 " - -b"i @ @@r1cLtjj||S)zApply Rotary Position Embedding (RoPE) to a single tensor. Interleaved layout: pair k uses adjacent elements [2k, 2k+1]. Args: x: Input tensor freqs_cis: Precomputed frequency tensor Returns: Transformed tensor )r*r+r,rr%rgs r0rrs 9 " . .q) < <rs ++++++.-----//////666666 @????????? # # # \ %2FF |F <FF \ FFFF, %~HH |H <HH \ HHHH, %2 |  \ . cc |clcc c  c  5< %& cccc8 %~ nn nln,n n  n  \ nnnn@!%$(!%## |# |#L#L # < # < # , #{T!# <$ # \####P>> |> > 5< %&>>>>. %~RR R,RR \ RRRR0!%$(  | |<<  ,   {T!  \F"&  | L|    L4   5<u| 34P!%$$ $ $\$\ $  $ \ $ , $$ \$$$$X!%  | \\ L  ,     \4A A A|A 5< %& AAAA(= |=|= \====$L L L|L 5< %& LLLL0H |H|H \HHHH6$49$$$$$#$$t$$$$" &c & & & & & &r1