9jo SSKrSSKrSSKrSSKrSSKrSSKJr SSKJrJ r J r SSK J r SSK r SSK Jr SSKrSSKrSSKJrJr SSKJr SSKJr SSKJrJr S S KJrJrJ r J!r! \RD"S S 5r#\RH"\#S 9RKS5r&Sr'Sr(Sr)Sr*"SS5r+Sr,S"Sjr-Sr."SS5r/SS/SS4S\0\1\2\14S\1S\2\3S\4S \44 S!jjr5g)#N)stderr)OptionalDictUnion)tqdm)infer_auto_device_mapdispatch_model)get_balanced_memory)snapshot_download) load_file save_file) get_codecget_32bit_codecget_lutsencode_weightsdfloat11z decode.ptx)pathdecodez0.5.0)i)encoded_exponent sign_mantissac,\rSrSrSr0r\S5rSrg) TensorManager4z Static utility class that manages tensor allocation and reuse to minimize memory allocation overhead during tensor reconstruction. c[U[5(a[R"U5nU[R ;a[[R UnUR 5U:aUSU$[R U [RR5 [R"U[RUS9n[SUSU3[S9 U[R U'U$)a Get a bfloat16 tensor with at least n_elements on the specified device. If a tensor already exists on the device and is larger than n_elements, a slice of the tensor with exactly n_elements is returned. If n_elements exceeds the size of the existing tensor, the existing tensor is deallocated and a larger one is allocated. Args: device: The device to allocate the tensor on (e.g., 'cuda:0') n_elements: The exact number of elements required Returns: A bfloat16 tensor with exactly n_elements on the specified device Ndtypedevicez Allocated z bf16 on device file) isinstancestrtorchrr_tensorsnumelcuda empty_cacheemptybfloat16printr)r n_elementsexisting_tensor new_tensors K/home/wildlama/miniconda3/lib/python3.13/site-packages/dfloat11/dfloat11.pyallocate_bfloat16TensorManager.allocate_bfloat16<s$ fc " "\\&)F ]++ ++44V>&Q  :,&6vh?fM*4 v&N) __name__ __module__ __qualname____firstlineno____doc__r% staticmethodr0__static_attributes__r3r2r/rr4s  H''r2rc.^^[T5mUU4SjnU$)a Creates a PyTorch forward pre-hook that decodes compressed DFloat11 weights on-the-fly. This hook reconstructs full-precision weights from compressed representations using a custom CUDA kernel during the forward pass. Args: threads_per_block: CUDA thread configuration bytes_per_thread: Number of bytes processed per CUDA thread Returns: A forward pre-hook function for PyTorch modules c>URRn[US5(amURR 5HOup4[X5(a[ X5RU:XaM0UR X4RUSS95 MQ URR5nURR5nURRSn[RX%5n[[R "UTST-- 554n ["R$R'UR(5 [+U TUR,URR/5URR/5URR/5UR0R/5UR2R/5UR/5XvU/ S9 SSS5 [5U[6R85(a,UR;UR<UR>5Ul O[5U[6RB5(a,UR;URDURF5Ul Oj[HRJ"XRL5n [OURPU 5H0upU R;U R<U R>5U l M2 [US5(aKURRS5H,n[X5(dM[ X5n [UX5 A M. gg!,(df  GNl=f)Noffloaded_tensorsT) non_blockingrgridblock shared_memargs)+lutsrhasattrr=itemsgetattrregister_buffertorr&rshaperr0intmathceilcpr'Deviceindex_decodeshared_mem_sizedata_ptroutput_positionsgapsr"nnLinearview out_features in_featuresweight Embeddingnum_embeddings embedding_dimr$ tensor_splitsplit_positionszipweight_injection_moduleskeysdelattr)module_r tensor_nametensorr,n_bytesn_luts reconstructedblocks_per_gridweights sub_moduler[tmpbytes_per_threadthreads_per_blocks r/ decode_hookget_hook..decode_hookws## 6. / /'-'?'?'E'E'G# F00gf6R6Y6Y]c6c**; &W[ 8\] (H))//1 ))//1""1%&77K tyy4Ea4HK[4[)\]^aWW^^FLL ) 0AfNdNd $$&''002$$--/''002 $$&&&(l * fbii ( ()..##V%7%7FM - -)..%%v';';FM ((8N8NOG&)&*I*I7&S" $*KK 0G0GI_I_$` !'T 6. / /%77<<> 6//!&6CF0 ? 07* )s B%M M#)tuple)rqrprrs`` r/get_hookrugs/07r r2Tc  USnUSnUSn U(d>[R"U5V s/sHoRS5(dMU PM sn OU/n Sn U(aU S- n U(aU S- n U S- n [XS 9GHn U(d[RR X5OU n[ U5nUR5GH4unnUUR5;GaU[UR55;a[UR55UnURUR:XaURRU5 M[S US URS UR3[S 9 M[UR!55UnURUR:XaURU5 GM[S US URS UR3[S 9 GM1UR#S5nUn[%USS5H8unn['UU5(a[)UU5nM%[SU3[S 9 GM USS:Xa[+USUR-55 OU(aUbUS:aUS[.;a|['US5(d [+US05 U(aUR15OUUR2US'Ub1US:a+[5UR25[5[.5:XaUS-nOUR7USU5 USS:XGaOUR9[;Xx55 U R5GHunn[<R>"USR USS55(dM7[AU[BRD5(aURFn[IUS5 AMq[AU[BRJ5(aURFn[IUS5 AM[+US/5 UH_nUR#S5nUnUHn[)UU5nM URFn[IUS5 AURLROU5 Ma GM GMUSS:XdGMURQ[RRT5RW5n[+USUSS-S-USSUSS- RY5R[5S--5 GM7 GM U$s sn f)a{ Loads DFloat11 compressed weights from safetensors files and configures the model to use them with on-the-fly decompression. Args: model: The PyTorch model to load weights into directory_path: Path to the directory containing safetensors files dfloat11_config: Configuration for DFloat11 compression Returns: The model with configured DFloat11 compression rqrp pattern_dict .safetensorszLoading DFloat11 safetensorsz (offloaded to CPUz, memory pinned))desczShape mismatch for z: model z vs loaded r .NzCannot find module path for r`rr=rrr[rbrTrR).oslistdirendswithrrjoinr rF state_dictdictnamed_parametersrJdatacopy_r+r named_bufferssplit enumeraterErGsetattrtolistoffloaded_tensor_names pin_memoryr=lenrHregister_forward_pre_hookrure fullmatchr"rVr\r[rdrWrbappendrXr$uint32numpymaxitem)modeldirectory_pathdfloat11_config cpu_offloadcpu_offload_blocksrfrom_single_filerqrprwfsafetensors_files loading_desc file_name file_pathloaded_tensorsrg tensor_valueparambufferpartsreipartpattern attr_namesro attr_pathtargetpoutput_positions_nps r/load_and_replace_tensorsrs*((;<'(:;'7L ::n--aN1K-$2#32L,,  - -L +? CSBGGLL;Yb #9-*8)=)=)? %Ke..00$u'='='?"@@ !7!7!9:;GE{{l&8&88 ((6 3K= U`amasas`tu}CD"%"5"5"78EF|||'9'99 \2 3K=Vabnbtbtauv~DE$))#. )s4GAtvt,,!(!6 <[MJQWX 5Ry$55(9<;N;N;PQ&,>,FJ\_`J`fklnfotJgJ#*63F#G#G '0CR H_i,BYBYB[o{F44U2Y? 2 >EWZ[E[beflf~f~bDGH^D_c_ 2a 7 2#2259lKRy$6688BS9fg4@3E3E3G/GZ!||GSXXeCRj5IJJ#-fbll#C#C*0--C$+FH$=(+%/ %B%B*0--C$+FH$=(+%,F4NPR$S5? 090D1716A55(a SU ;aU SnO)[AU S5(a U RBnO [ES5e[GWU UXgXS9 U(dLSnURI5RK5HnUURL- nM [OSUS- SS3[PS9 U(aURSU5nU$US:XdS5e[UUUS5n[WXUS9n[YXUS9n[[X5n[]SUR_555(a[OS[PS9 U$!,(df  GN=f!,(df  GNn=f![:a nSnAGNhSnAff=f) a Load a model with DFloat11 compressed weights from local path or Hugging Face Hub. Args: dfloat11_model_name_or_path: Local path or HF Hub model name device: Target device for the model device_map: Strategy for distributing model across devices max_memory: Maximum memory allocation per device bfloat16_model: Optional pre-initialized model to load weights into cpu_offload: Enables CPU offloading; only keeps a single block of weights in GPU at once cpu_offload_blocks: Number of transformer blocks to offload to CPU; if None, offload all blocks pin_memory: Enables memory-pinning/page-locking when using CPU offloading from_single_file: Whether to load a single safetensors file pattern_dict: Dictionary mapping regex patterns to submodule lists **kwargs: Additional arguments passed to AutoModelForCausalLM.from_config Returns: Model with DFloat11 compressed weights configured for on-the-fly decompression zeExpected `dfloat11_model_name_or_path` to be the path to a safetensors file, but found a directory: "".z The file "z" does not exist./__) local_dirrversionrqrprw config.jsonrutf-8encodingNr)AutoModelForCausalLM AutoConfigGenerationConfig)no_init_weights torch_dtypezd"dfloat11_config" not found: it is expected to be found in the config file or passed as an argument.)rrrrzTotal model size: geAz0.4fz GBr autoz>device_map should be 'auto' if no specific device is provided.rw)rno_split_module_classesc3R# UHoRRS:Hv M g7f)cpuN)rtype).0rs r/ 0DFloat11Model.from_pretrained..sN;M%<<$$-;Ms%'zqWarning: Some model layers are on CPU. For inference, ensure the model is fully loaded onto CUDA-compatible GPUs.)0rrisfileisdirIsADirectoryErrorFileNotFoundErrorexistsreplacer rrqrpopenrjsonload transformersrrrtransformers.modeling_utilsrfrom_pretrained from_configr$r* tie_weightsevalgeneration_config Exceptionr"rrErAttributeErrorrrvaluesnbytesr+rrIrr rr any parameters)clsrrrrbfloat16_modelrrrrrwkwargsdfloat11_model_pathconfigrrrrrrrer model_bytesrrs r/rDFloat11Model.from_pretrainedVs)F ww~~9::&A#:;;'+PQlPmmo)pqq'*5P4QQb(cddww~~9::&A#&A&I&I#t&T#ww~~&9::%&Aa %#*->,<(4 ("'',,':MJCZabfg!YYq\Fc#E W W C //0CDF ",88(-:@!!# # $4$D$DEX$Y!*;' fd # #(9V(C$%67O V. / /$44O "HI I ! &#! K))+224u||+ 5 &{S'8&>cB P HHV$E ' i)i i'3E?>;Z[ ,UcstJ.ueuvJ"55EN5;K;K;MNNNJQWX }cb#"  s+$L.AL;L+ L L(+ M;Mrc 2URUUUUUUUU SUS9 $)NT) rrrrrrrrrrw)r) rrrwrrrrrrrs r/rDFloat11Model.from_single_files:""(;!!)#1!!%#  r2r3) NrNNFNTFN)NrNFNT)r4r5r6r7r8 classmethodr#rrrrKboolrlistrrr:r3r2r/rrPs !% GK!,0!&7;{%({ { { T%S/5c?"BCD {{%SM{{{tCcN34{{| !% GK!,0   3S >*     T%S/5c?"BCD  %SM   r2ri'rw save_path block_rangesave_single_filecheck_correctnessc*^4[R"USS9 SnSnUR5GHupUR5GHup[R "X5(dM#U(dDSU ;a2U R SS5upURU 5n[XS5 O [X S5 US- nXcS::aM}XcS:aSn M/n[U [R5(aU RRR[R :Xd+SU SU RRR35eUR#U RRR%5R'5R)55 [+U S 5 GO[U [R,5(aU RRR[R :Xd+SU SU RRR35eUR#U RRR%5R'5R)55 [+U S 5 OU HnUR/S5nU nUHn[1UU5nM URRR[R :Xd.SU SUSURRR35eUR#URRR%5R'5R)55 [+US 5 M [3[R4"U55unn[7U5unnnUR95 [;U5n[=UU[>[@S5unnnnnU(Ga8[RB"S 5m4URDSnURG5n URG5n![I[KU44S jUUUUU/55un"n#n$n%n&[RL"U [R T4S 9n'[O[PRR"U![@S[>-- 554n(URU[RV5RY5n)[@SS -S -U)SSU)SS- R[5R]5S--n*[_SU*S35 [`RbReT4Rf5 [iU([@U*U"Rk5U#Rk5U$Rk5U%Rk5U&Rk5U'Rk5UU!U / S9 SSS5 [R4"U5U'R'5:HRm5R]5n+U+(a [_S5 O[oSU S35eU RqSU5 U RqSU5 U RqSU5 U RqSURU[Rr55 U RqSU5 U RqSU5 U(aGMgU Ru5n,U,R5V-V.s0sHun-n.U SU-3U._M n,n-n.[wU,[RxR{X*R}SS5S-55 GM GM U(Ga[~[@[>US.n/[US5(aU/URlB[US 5(aU(dURU5 O8[wURu5[RxR{US!55 Sn0[RxR[RxR{US"55(ah[[RxR{US"5S#S$S%9n1[R"U15n2S&U2;a[U2S&[5(aSn0SSS5 U0(aL[[RxR{US"5S'5n3[R"S&U/0U3SS(9 SSS5 ggg!,(df  GN)=fs sn.n-f![a GNf=f!,(df  N=f!,(df  g=f))NT)exist_okrr{rFz@Expected weights to be in bfloat16 format for compression, but 'z ' has dtype r[zcuda:0c&>URT5$)N)rI)xrs r/ compress_model..7suvuyuy{AvBr2rr}r|r~zUsing z bytes of shared memory.r?uN✅ Correctness check passed: decompressed weights match the original weights.ue❌ Correctness check failed: The decompressed weights do not match the original weights for module "rrDrrrTrUr`rfrxrrsave_pretrainedzmodel.safetensorsrrrrrw)indent)KrmakedirsrFrrrrsplit get_submodulerr"rVr\r[rrr$r*rdetachrflattenrdrWrrGrcatrprint_code_tablerrrprqrrJr&rmapr)rKnprMrXrrrrr+rNr'rOrPrQrSall RuntimeErrorrHuint8rr rrrrrErrrrrrrrrdump)5rrwrrrr block_index save_modelrrrrn parent_name child_nameparentrmrrrr_codec_countercodecrftablerDencoded other_8bitsrTrUr`rjr,ri cuda_luts cuda_encodedcuda_other_8bitscuda_output_positions cuda_gaps cuda_outputsrlrrR _is_correctrkeyvaluer save_configrr config_filers5 @r/compress_modelr)s_KK D)KJ+113%*%8%8%: !I||G//'i'2;2B2B32J/ !&!4!4[!AD9$7q a.0 q>1!&Jj",,77%,,11775>>IQZ[dZeeqr|sDsDsIsIsOsOrPQQINN:#4#4#9#9#@#@#B#F#F#H#P#P#RSJ1 BII66%,,11775>>IQZ[dZeeqr|sDsDsIsIsOsOrPQQINN:#4#4#9#9#@#@#B#F#F#H#P#P#RSJ1&0 ) 4!+!&A%,VQ%7F"' &}}11775>>I]^_h^iijktjuvBCICPCPCUCUC[C[B\]]Iv}}'9'9'@'@'B'F'F'H'P'P'RS1&0$-UYYw-?#@ "1(";q%&&(P^_fhmoBSTUBVQWM&6o$"\\(3F!ZZ]F!,!2!2!4J%mmoGbfgjlBEIKRT_aqswDxhycz_I|-=?TV_#(;;zX^#_L'*2777>OPQ>RUe>e3f+g'h&kO*:*?*? *M*S*S*U'&7&:Q&>&BFYZ[Z\F]`stwuw`xFxE}E}EFEFEFGJKFK'KOF?"33KLM 5_ , 5+ , ,5E  ! !) , e&&("'',,yBU*V W 77>>"'',,y-@ A Abggll9mWY]3^3^"'K U bggll9mr8s (( <0-2PP  * *:| D ,,H % 2 28 <  00fI`AH2] ] H Z!" L*sDI~&L*L*c L*  L*  L*r2