3j}XSrSSKJr SSKrSSKrSSKJr SSKJr SSK J r SSK r SSK r SSKJrJrJrJr \ (aSS KJr \R("\5rS rS rS r1S krS$SjrS%SjrS&SjrS'SjrS(SjrSSS.S)Sjjr S*Sjr!S+SSS.S,Sjjjr"S-Sjr#S+SSS.S.Sjjjr$S/Sjr%"SS\5r&S0Sjr'\"\ RPRRRTRV\&5S5r,S1S jr-\"\ RPRRR\RV\&5S!5r/\"\ RPRRR`RV\&5S"5r1\"\ RPRRRdRV\&5S#5r3g)2a~SVDQuant W4A4 quantization layout for tensor cores. Each quantized linear stores: qweight: (N, K // 2) int8 packed W4 residual scale=wscales: (K // 64, N) bf16/fp16 per-group weight scales proj_down: (K, R) bf16/fp16 SVD down projection (V^T) proj_up: (N, R) bf16/fp16 SVD up projection (U) smooth_factor: (K,) bf16/fp16 input-side smoothing LoRA-style proj_down / proj_up recover the outlier-heavy singular directions that pure 4-bit quantization cannot represent; the dispatched kernel fuses activation quantization + low-rank correction + int4 matmul into a single call. ) annotationsN)Sequence) dataclass) TYPE_CHECKING)BaseLayoutParamsQuantizedLayoutdequantize_argsregister_layout_opQuantizedTensor@g?>0noofffalsec[R"U5nUcU$UR5R5[;$N)osgetenvstriplower_FALSE_ENV_VALUES)namedefaultvalues \/home/wildlama/miniconda3/lib/python3.13/site-packages/comfy_kitchen/tensor/svdquant_w4a4.py _env_enabledr,s4 IIdOE } ;;=   (9 99cL[[RRSS5$)Nbackend_override)getattrckregistry _thread_localr r_registry_backend_overrider(3s 2;;,,.@$ GGr c[SSS9(dg[5nUbUS:wagUR(aUR(dgURUR:wagSSKJn [USS 5(dgU$![ a gf=f) aReturn the CUDA backend module when SVDQuant can safely bypass registry. ComfyUI currently disables comfy_kitchen's global CUDA backend on PyTorch CUDA < 13, but this SVDQuant extension is built locally and works on the cu128/RTX 5090 environment. The QuantizedTensor path can call the CUDA implementation directly while still respecting an explicit non-CUDA backend override such as ``with ck.use_backend("eager")``. "COMFY_KITCHEN_SVDQUANT_DIRECT_CUDAT)rNcudar)r+_EXT_AVAILABLEF)rr(is_cudadevicecomfy_kitchen.backendsr+ Exceptionr#) input_tensorqdataoverride cuda_backends r_direct_cuda_backendr57s 0svdquant_w4a4_can_share_quant..xsGwV(00wsrNT) lenallr=r< act_unsignedrE smooth_factor proj_downdata_ptrtorchequal)weightsrGrHr9rLparamss rsvdquant_w4a4_can_share_quantrYcsB& 7|q GwG G G 1:  D!"+ ## $T->->(? ?$V%9%94;M;MNN$V%5%5t~~FF    ( ( *d.@.@.I.I.K K    % % '4>>+B+B+D D   {{6//1C1CDD{{6++T^^<<'( r c  [RU5upxpn [[URSS55n UbUR XXX+UU S9n O[ R "XXX+UU S9n [RU5nU SUU4$)NrQFactwgtascaleswscales lora_act_inlora_upbiasrQ)r7get_plain_tensorsr<r#r=scaled_mm_svdquant_w4a4r$get_out_features_from_storage)q_xr^lora_act weight_qtrbr4mr2r__smooth _proj_downproj_uprQout out_featuress r'_w4a4_forward_from_quantized_activationros4P3a3abk3l0EG 1 1>5IJL22 %3  (( %  0MMeTL r7L  r )validate_shared_quantassume_shared_quantc 2[U5S:XagUcS[U5-n[U5[U5:wa$[S[U5S[U5S35e[XUS9(d [S 5eUSn[R U5upgpn [ [ URS S 55n URn URS U S 5n U RSnU (a U [-nU nOU nSn[X5nUbURUUU U US S9unnnO[R"UUU U US9unnn/n[XS S9H>unn[UUUUUUUS9unnUR!UR"/U SS QUP765 M@ [#U5$)aRun split SVDQuant linears while sharing activation quantize. This is intentionally a runtime grouping helper, not a fused-QKV storage format. It preserves split Q/K/V checkpoint tensors and only removes the repeated per-input work that is identical across those projections. rr'Nrgot weights but  biasesrF5SVDQuant weights do not share quantization parametersrQFTsmooth lora_downrQlora_xreuse_workspaceryrzrQr{strict)ri)rO ValueErrorrYr7rcr<r#r=rAreshape_GELU_UNSIGNED_SHIFTr5quantize_svdquant_w4a4r$ziproappendtuple)r1rWbiasesrprqfirstr2_wscalesryrS_proj_uprQ orig_shapex2drix_mainr{r4rfr^rgoutputsrLrbout2drns rsvdquant_w4a4_grouped_linearrs 7|q ~3w<' 6{c'l"4G ~]3v;-wOPP (7J PQQ AJE3O3a3abg3h0EV ~uEFL##J   r:b> 2C ! A++'6L!-!D!D % "E" Wh"$!:!: % " WhGGD9 E (FD,! | u}}Dj"oD|DE : >r c0^USR5m[U4SjU55(a [S5eTS;a,[R"[ U5SS9R 5$[R"[ U5US9R 5$)Nrc3H># UHoR5T:gv M g7frdim)rKtrs rrM'_cat_svdquant_n_axis..s +7a557c>7s"z6cannot fuse mixed natural/tile-packed SVDQuant tensors>r)ranyrrUcatr contiguous)tensors natural_dimrs @r_cat_svdquant_n_axisrss !*.. C +7 +++QRR f}yywQ/::<< 99U7^ 5 @ @ BBr c JSSKJn [U5S:Xa [S5eUcS[U5-n[U5[U5:wa$[S[U5S[U5S 35e[ XUS 9(d [S 5eUSnUR nUR Sn[UR5n/n /n /n /n UHn [RU 5upnnnU R R SU:wa [S 5e[U R R5U:wa [S 5eU R[RU55 U RU5 U RU5 U RU5 M [U SS9n[U SS9n[U SS9n[SU55(aSnO/n[XSS9HDunnUc*[ R""UUR$UR&S9nURU5 MF [ R("[+U5SS9R-5n[R/UUR0[3U 5U4UR4UUR6US9nU"USU5U[+U 54$)aFuse split SVDQuant projections into a runtime-only wide projection. This does not change checkpoint storage. It concatenates the already loaded split weights along output-N so a caller can execute Q/K/V as one wider SVDQuant linear and split the output view afterwards. rr rzexpected at least one weightNrrsrtrurFrvz.-s +FD4> $ $Q '; 6[\ \ ++ , <^_ _8VVW\]^ e f 'v1=K&v1=K(qAM +F +++  6=GD!|{{1M,?,? H\H\]   d #>YYuZ0a8CCE ) 0 0** %{3(("00!1F  %CVL l r c[U5(d [S5e[XU5n[[R "U[U5SS95$)zGExecute a runtime-fused split projection and return split output views.zCfused_weight must be a TensorCoreSVDQuantW4A4Layout QuantizedTensorrwr)r? TypeError _w4a4_forwardrrUsplit)r1 fused_weightroutput_featuresrms r"svdquant_w4a4_fused_grouped_linearrGsD %\ 2 2]^^  J ?C S%"8bA BBr c\rSrSrSrSrSr\"SS9"SS\55r \ SS j5r \ SS j5r \ SS j5r \ SS j5r\ SS j5rSrg)r7iTufSVDQuant W4A4 weight quantization with low-rank correction. Note: Offline-quantized only — `quantize()` raises NotImplementedError because SVDQuant factorization requires calibration (smooth_factor, proj_down, proj_up) that must be computed from activation statistics. Use the DeepCompressor pipeline to produce the pre-quantized tensors. )rFT)frozencd\rSrSr%SrS\S'S\S'S\S'SrS\S 'SrS\S 'SS jrS r S r g)#TensorCoreSVDQuantW4A4Layout.Paramsija SVDQuant W4A4 parameters. Inherits `scale` (= wscales), `orig_dtype`, `orig_shape` from BaseLayoutParams. Adds the three tensors that parameterize the low-rank correction and input smoothing, plus a logical-transpose flag used by the aten.t / aten.mm dispatch path. torch.TensorrSrlrRFr<rQr8c /SQ$)NrrSrlrRr'selfs r_tensor_fields2TensorCoreSVDQuantW4A4Layout.Params._tensor_fieldsysE Er cgrr'rs r_validate_tensor_fields;TensorCoreSVDQuantW4A4Layout.Params._validate_tensor_fields|s r r'N)returnz list[str]) __name__ __module__ __qualname____firstlineno____doc____annotations__rQr8rr__static_attributes__r'r rrrjs:  ##" d" D  F r rc [S5e)NzvSVDQuant W4A4 requires offline calibration (DeepCompressor). Load pre-quantized tensors via `from_state_dict` instead.)NotImplementedError)clsr>kwargss rquantize%TensorCoreSVDQuantW4A4Layout.quantizes " H  r c jURSnURnURn[R"X5US9n[ R "XbRURS9upxn [ R"XqXRXRSS9SUn U R5R5$)u%Reconstruct the effective weight W_eff such that plain ``x @ W_eff.T + bias`` reproduces the SVDQuant kernel output to bf16 precision. Uses the kitchen kernel itself with an identity input rather than reimplementing dequant in Python. Kitchen weight layout is natural row-major packed int4 ``(N, K/2)`` — the kernel reads it directly, so this path stays bit-exact with the actual compute path regardless of per-group scaling / LoRA composition details. Tile-packed storage is handled by the same kernel path and produces the same logical weight. rr)ryrzN)r\r]r^r_r`rarb)rr.rrUeyer$rrRrSrdrrlrr) rr2rXrr.rBrrfr^rgw_effs r dequantize'TensorCoreSVDQuantW4A4Layout.dequantizes''* !!ii @!#!:!: ,,8H8H" h** ..t  ;wwy##%%r cURnURURURURUR 4$r)r=_qdatarrRrSrl)rqtensorps rrc.TensorCoreSVDQuantW4A4Layout.get_plain_tensorss2 OO~~qwwaiiOOr cUR5S:Xa[URS5[-$[URS5$)Nrr)rintrA_TILE_PACKED_BLOCK_N)rr2s rre:TensorCoreSVDQuantW4A4Layout.get_out_features_from_storages< 99;! u{{1~&)== =5;;q>""r cbUURURURURS.$)zSerialization mapping. Suffixes compose onto the owning Parameter's key (typically `*.weight`), producing for example `transformer_blocks.0.attn.to_q.weight`, `...weight_scale`, `...weight_proj_down`, etc. )_scalerkr_smooth_factorr)rr2rXs rstate_dict_tensors/TensorCoreSVDQuantW4A4Layout.state_dict_tensorss1ll **$22   r r'N)r>rrztuple[torch.Tensor, Params])r2rrXrrr)rr rzKtuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor])r2rrr)r2rrXrrzdict[str, torch.Tensor])rrrrrMIN_SM_VERSIONQUANTIZES_INPUTrrr classmethodrrrcrerrr'r rr7r7TsN Od!,   %   &&2P%P TPP ##     r r7c @[RU5up4pVn[[URSS55nUR n UR SU S5n U R Sn U(a U [-n U n OU n Sn [X5nUb-URU UUUU SS9unnnURXUUUXrUS9nO5[R"U UUUU S 9unnn[R"XUUUXrUS9n[RU5nUSU R "/U SSQUP76$) u,Compute y = x @ W^T + bias via the int4 kernel. For layers flagged ``act_unsigned`` (nunchaku convention: post-GELU fc2), we apply the +0.171875 shift to the main-path activation here at the layer so it falls into the unsigned [0, 15] quantization grid. LoRA continues to use the raw un-shifted activation (SVDQuant invariant: LoRA is the residual between full-precision W and the int4 approximation, computed on the pre-quantization activation). Kernel API stays shift-free — shift is a Qwen/Flux model-topology constant, not a quantize-op parameter. rQFrwrNTrxr[r}) r7rcr<r#r=rArrr5rrdr$re)r1rhrbr2r_ryrSrlrQrrrirr{r4rfr^rgrmrns rrrsd"2N1_1_`i1j.EFw 1 1>5IJL##J   r:b> 2C ! A++'6L!-!D!D % "E" Wh22 '%3  "$!:!: % " Wh(( '%  0MMeTL r7?? :JsO :\ ::r c`SSKnSSKJn USn[XT5(d4[R R RR"U0UD6$URnURUURSURS4UR(+S9nU"URSU5$)uZero-copy logical transpose — flip the ``transposed`` flag. Lets ``F.linear(x, W)`` decompose into ``x @ W.t()`` without reordering any storage; ``mm`` / ``addmm`` handlers below unwind the flag. rNrr )rr8r7) dataclassesr9r r:rUopsatenrrr=replacerr8r)qtargsrrr r1old new_paramss r_handle_w4a4_trs%7L l 4 4yy~~''888   C$$ NN1%s~~a'89~~%%J <..0NPZ [[r cRURR(d [S5eU$)zDReturn rhs unchanged if it is logically transposed (represents W^T).z\SVDQuant W4A4 GEMM expects the RHS to be W.T (stored W). Use F.linear(x, W) or mm(x, W.t()).)r=r8 RuntimeError)rhss r_resolve_svdquant_rhsrs( ;; ! ! 2   Jr cSSKJn USUSpT[U5S:aUSOSn[XS5(d2[R R R"[XEU456$[XC5(aUR5nURR(a8[R R RXER5U5$[XEU5$)u3Direct F.linear(input, W, bias) → kitchen kernel.rr rN) r9r rOr:rUnn functionallinearr rr=r8r)rrrr r1rLrbs r_handle_w4a4_linearr's&7DG&$i!m47D f . .xx""))?LRV;W+XYY,00#..0  ~~  xx"")),8I8I8KTRR t 44r cSSKJn USUSpT[XS5(d[R"[ XE456$[XC5(aUR 5n[U5n[XESS9$)uoHandle ``mm(x, W.t())`` — the decomposition F.linear takes when the weight is a non-default tensor subclass. rr rNrb) r9r r:rUmmr rrr)rrrr rCrDs r_handle_w4a4_mmr8sd & 7DGq a ) )xx!011!%% LLNa A D ))r cSSKJn USUSUSpen[Xc5(d[R"[ XEU456$[XS5(aUR 5n[U5n[XVUS9$)z!Handle ``addmm(bias, x, W.t())``.rr rrr) r9r r:rUaddmmr rrr)rrrr rbrCrDs r_handle_w4a4_addmmr Hsn&a$q'47QD a ) ){{OTaL9::!%% LLNa A D ))r )rstrrr<rr<)rz str | None)r1rr2r)r>rrr<)rCrrDrrr<)rWSequence[torch.Tensor]rGr<rHr<rr<)rfrr^rrgrrhr rbtorch.Tensor | Nonerirrztuple[torch.Tensor, int]r) r1rrWr r$Sequence[torch.Tensor | None] | Nonerpr<rqr<rtuple[torch.Tensor, ...])rr rrrr) rWr rrrpr<rqr<rzrs # $! XX%   8 $/:H<N - #--  -  -`! ! !! !  ! !!>48A #( % AA #A 1A  A  AAHC48H#( % H #H 0H H  H B HV C C C$ C# C  Cm ?m d<;<;<; <; <;~EIINN$$,,.JK\L\.EIINN))113OP 5Q 5 EIINN%%--/KL *M *EIINN((002NO *P *r