9j BSSKrSSKrSSKrSSKrSSKrSSKrSSKJr SSKJ r SSK J r SSK J r SSKJrJr SSKrSSKJr S S KJrJrJrJr S S KJr S S KJrJr S S KJrJ r J!r! S SK"J#r# \"\$5r%Sr&Sr'S2Sjr(S2Sjr)Sr*S3Sjr+S3Sjr,S4Sjr-S3Sjr.S5S\/S\/S\04Sjjr1S6S\/S\/S\0S\04Sjjr2S\RfRhS \Rj4S!jr6S7S\RfRhS"\7S#\04S$jjr8S%\RrRtS&\74S'jr;S\RfRh4S(jrS\RfRhS)\\RfRh/\04S-4S,jr?S-r@S.\7S)\74S/jrAS0\\ \RfRh\/4S)\B\RfR4S1jrDg)8N) defaultdict)Iterable) nullcontext)Path)CallableUnion) get_logger)FSDP_MODEL_NAMEOPTIMIZER_NAMESAFE_WEIGHTS_NAME WEIGHTS_NAME)get_module_class_from_name)get_non_persistent_buffers is_peft_model)get_module_children_bottom_upis_compiled_modulesave)is_torch_versioncxS[R;aS[RS'S[RS'g)zS Enables RAM efficient loading of Hugging Face models for FSDP in the environment. ACCELERATE_USE_FSDPTrueFSDP_CPU_RAM_EFFICIENT_LOADINGNosenvironU/home/wildlama/miniconda3/lib/python3.13/site-packages/accelerate/utils/fsdp_utils.py!enable_fsdp_ram_efficient_loadingr!'s. BJJ.,2 ()39BJJ/0rc*S[RS'g)zT Disables RAM efficient loading of Hugging Face models for FSDP in the environment. FalserNrrrr "disable_fsdp_ram_efficient_loadingr$1s4;BJJ/0rcU(a&[U5(aSSKJn U"XRS9$Ub SSKJn U"XS9$UR 5$)Nr)get_peft_model_state_dict adapter_name)get_model_state_dictoptions)rpeftr&active_adapter'torch.distributed.checkpoint.state_dictr) state_dict)model adapter_only sd_optionsr&r)s r _get_model_state_dictr38sH e,,2(=Q=QRRP#E>>!!rcU(a'[U5(aSSKJn U"XURS9$Ub SSKJn U"XUS9$UR U5$)Nr)set_peft_model_state_dictr')set_model_state_dictr*)rr,r5r-r.r6load_state_dict)r0r/r1r2r5r6s r _set_model_state_dictr8GsN e,,2(I]I]^^P#EzJJ$$Z00rc SnURS:XaUSSKJn SSKJn U"UR UR :H[URSS5[URSS5S9nU$) Nr r)StateDictOptions StateDictTypeoffload_to_cpuF rank0_only)full_state_dict cpu_offloadbroadcast_from_rank0) fsdp_versionr.r:2torch.distributed.fsdp.fully_sharded_data_parallelr<state_dict_typeFULL_STATE_DICTgetattrstate_dict_config) fsdp_pluginr2r:r<s r _prepare_sd_optionsrIVsjJ1$LT%'77=;X;XX = =?OQVW!()F)F V[!\ rcSSKJs Jn SSKJn SSKJn SSKJn [R"USS9 URU R:Xa/URS:n XRlXRlUR S:Xa1URX RURUR"5O [%5n ['U5n U  [)X%U S9n URU R:XaUS:Xa [*S 3O [*S US 3n[R,R/X>5nUR0S:XaF[2R5S U35 [6R8"X5 [2R5S U35 GOfURU R::XaUS:Xa[*S UR0S 3O[*S US UR0S 3n[R,R/X>5n[2R5S U35 [6R8"X5 [2R5S U35 OURU R<:Xa[R,R/U[*S U35n[R"USS9 [2R5S U35 SU 0n UR9U UR?U5U"5S9 [2R5S U35 SSS5 g!,(df  g=f)NrDefaultSavePlannerFullyShardedDataParallelr;Texist_okr r1r2.bin_zSaving model to zModel saved to _rankr0r/storage_writerplanner) torch.distributed.checkpoint distributed checkpoint,torch.distributed.checkpoint.default_plannerrLrCrNr<rmakedirsrDrE num_processesrGr=r>rBoptim_state_dict_configrrIr3r pathjoin process_indexloggerinfotorchrLOCAL_STATE_DICTSHARDED_STATE_DICTFileSystemWriter)rH acceleratorr0 output_dir model_indexr1dist_cprLFSDPr<is_multi_processctxr2r/ weights_nameoutput_model_fileckpt_dirs r save_fsdp_modelrrgs22OcPKK T*""m&C&CC'44q87G%%43C%%0  # #q (  .. 0M0M{OrOr ] %[1J *5Xbc  & &-*G*G G7Ba7Go.d3P_O``abmannrMsL " Z F ((A- ./@.ABC :9 o.?-@AB  ( (M,J,J J!###5)B)B(C4H'(+eK ? JJz 5 KK/*;)<= >  ( (M,L,L Lww||J?2C1[M0RSH KK4 0 KK*8*5 6!:.J LL%&77A*,   KK/(4 5? s HK88 LcSSKJs Jn SSKJn SSKJn SSKJn UR5 URU R:Xa/URS:n XRl XRlURS:Xa1URX RURUR 5O [#5n [%U5n U  URU R:XGa['U5ULaWUR(S:waGUR*(d6UR,(dURS:Xa [/S5eSSS5 gUS:Xa [0S3O [0SUS3n [2R4R7X=5n[8R;S U35 UR*(+=(d UR<nU(a[>R@"US S 9nO0n[8R;S U35 GOjURU RB:XaUS:Xa[0S UR(S3O[0SUS UR(S3n [2R4R7X=5n[8R;S U35 [>R@"US S 9n[8R;S U35 OURU RD:Xa[0U;a)[2R4R7U[0SU35OUn[8R;S U35 S[GX%U S90nURAUURIU5U"5S9 USn[8R;S U35 [KUWX\S9nSSS5 U$!,(df  W$=f)Nr)DefaultLoadPlannerrMr;r zzSet the `sync_module_states` flag to `True` so that model states are synced across processes when initializing FSDP objectrRrSzLoading model from T weights_onlyzModel loaded from rTr0rQ)r/storage_readerrW)&rXrYrZr[rtrCrNr<wait_for_everyonerDrEr]rGr=r>rBr^rrItyperais_fsdp2sync_module_states ValueErrorr rr_r`rbrcis_main_processrdloadrerfr3FileSystemReaderr8)rHrhr0 input_dirrjr1rkrtrlr<rmrnr2roinput_model_file load_modelr/rq load_results r load_fsdp_modelrsO22OcP!!#""m&C&CC'44q87G%%43C%%0  # #q (  .. 0M0M{OrOr ] %[1J  & &-*G*G GE{$&;+D+D+IR]RfRf"55+:R:RVW:W$3 8Ca7Go.d3P_O``abmannrMsL!ww||ID  KK-.>-?@ A(111P[5P5PJ"ZZ(8tL  KK,-=,>? @  ( (M,J,J J!###5)B)B(C4H'(+eK-?@ A$44HJ KK,-=,>? @  ( (M,L,L L&&y8 Y?*;1[M(JK  KK-hZ8 9!#8fp#qrJ LL%&77A*,   $G,J KK,XJ7 8+E:Lp [ \ ] \ s A8M<H$M<< N c&SSKJs Jn SSKJn SSKJn SSKJn [R"USS9 URS:Xa1URX0RURUR5O [5n [U5n U  URS:XaSS KJn U "X2U S 9n OUR%X25n URU R&:XaUR(S:XaUS:Xa [*S 3O [*S US 3n[R,R/XN5n[0R3S U35 [4R6"X5 [0R3SU35 O[R,R/U[*S U35n[R"USS9 [0R3S U35 UR7SU 0UR9U5U"5S9 [0R3SU35 SSS5 g!,(df  g=f)NrrKrMr;TrOr r get_optimizer_state_dictr*rRrSzSaving Optimizer state to zOptimizer state saved in optimizerrU)rXrYrZr[rLrCrNr<rr\rBrDrGr^rrIr.roptim_state_dictrErar r_r`rbrcrdrrg)rHrhrr0rioptimizer_indexrkrLrlr<rnr2r optim_stateoptim_state_nameoutput_optimizer_filerqs r save_fsdp_optimizerrs22OcPKK T*  # #q (  .. 0M0M{OrOr ] %[1J  # #q ( X25ZXK//AK  & &-*G*G G((A-/>!/C~&d+NK[[\]l\mmqIr!)+ Z(R% 89N8OPQ ;> 78M7NOPww||J>2B!OCT0UVH KK4 0 KK4XJ? @ LL'5&77A*,   KK3H:> ?5 s E0H HcSSKJs Jn SSKJn SSKJn UR 5 URS:Xa1URX0RURUR5O [5n [U5n U  URU R:XaSn URS:XdURR(dUS:Xa [ S3O [ SUS3n ["R$R'XM5n[(R+SU35 [,R."USS 9n [(R+S U35 O[ U;a)["R$R'U[ SU35OUn[(R+S U35 URS :XaSS KJn U"X2U S9n OUR5X25n SU 0n UR/U UUR7U5S9 U Sn [(R+SU35 URS:Xa"UR9X2U S9nUR;U5 O SSKJn U"X2XS9 SSS5 g!,(df  g=f)NrrMr;r rRrSzLoading Optimizer state from TruzOptimizer state loaded from zLoading Optimizer from r rr*r) checkpoint_idrwzOptimizer loaded from )r0optimr)set_optimizer_state_dict)rXrYrZrCrNr<rxrBrDrGr^rrIrErar>r rr_r`rbrcrdr~r.rrroptim_state_dict_to_loadr7r)rHrhrr0rrr1rkrlr<rnr2roptimizer_nameinput_optimizer_filerqr flattened_osdrs r load_fsdp_optimizerrs422cP!!#  # #q (  .. 0M0M{OrOr ] %[1J  & &-*G*G GK((A-[5X5X5c5c/>!/C~&d+NK[[\]l\mmqIr(*ww||I'N$ ;*:!O;L(MN  KK1(< =''1,\6uQ[\ "33EE & 4K LL&&77A   &k2K KK0 ; <  # #q ( 99it9uM  % %m 4 X $U{ WO s >GI  Icheckpoint_dir save_pathsafe_serializationchSSKJs Jn SSKJs Js Jn 0n[ U5nUR SS9 URUURU5UR5SS9 U(a U[- OU[- n[UR55S:XaU[U5Sn[XQUS9 U$)z Passthrough to `torch.distributed.checkpoint.format_utils.dcp_to_torch_save` Will save under `save_path` as either `model.safetensors` or `pytorch_model.bin`. rNTrO)rwrWno_distr )r)rXrYrZ)torch.distributed.checkpoint.format_utils format_utilsrmkdir_load_state_dictr_EmptyStateDictLoadPlannerrrlenkeyslistr)rrrrkdist_cp_format_utilsr/s r )_distributed_checkpoint_to_merged_weightsrRs32LLJYI OOTO"))//?$??A * 2D --UaIaI :?? "Z 0 34 3EF r output_pathremove_checkpoint_dirc[U5nSSKJn [SS5(d [ S5eUR 5(dUS- R 5nUS- R 5nSUS 3nU(aU(aUS - nUS US US 3- nUS- nO+U(aUS- nUSUS3- nOU(aUS- nUSUS3- n[ U5eU"5nUR (aq[RSU35 [XU5n [RSU 35 U(a.[RSU35 [R"U5 UR5 g)a Merge the weights from sharded FSDP model checkpoints into a single combined checkpoint. Should be used if `SHARDED_STATE_DICT` was used for the model. Weights will be saved to `{output_path}/model.safetensors` if `safe_serialization` else `pytorch_model.bin`. Note: this is a CPU-bound process. Args: checkpoint_dir (`str`): The directory containing the FSDP checkpoints (can be either the model or optimizer). output_path (`str`): The path to save the merged checkpoint. safe_serialization (`bool`, *optional*, defaults to `True`): Whether to save the merged weights with safetensors (recommended). remove_checkpoint_dir (`bool`, *optional*, defaults to `False`): Whether to remove the checkpoint directory after merging. r) PartialState>=z2.3.0z/`merge_fsdp_weights` requires PyTorch >= 2.3.0`pytorch_model_fsdp_0 optimizer_0zTried to load from z) but couldn't find a valid metadata file.zE However, potential model and optimizer checkpoint directories exist.zPlease pass in either z/pytorch_model_fsdp_0 or z /optimizer_0zinstead.z8 However, a potential model checkpoint directory exists.zPlease try passing in z/pytorch_model_fsdp_0 instead.z< However, a potential optimizer checkpoint directory exists.z/optimizer_0 instead.zMerging FSDP weights from z.Successfully merged FSDP weights and saved to z"Removing old checkpoint directory N) raccelerate.staterrr|existsr}rbrcrshutilrmtreerx) rrrrrmodel_path_existsoptimizer_path_existserrstaters r merge_fsdp_weightsrnsx(.)N- D' * *JKK  " "+.DDLLN!/-!? G G I#N#33\] !6 Z ZC +N+;;TUcTddpq qC : C  M MC +N+;;YZ ZC " Q QC +N+;;PQ QCo NE  00@AB=n[mn  DYKPQ KK<^[[5nURSS9H3up#[U5T;dMU[U5R U5 M5 T"U5nUR 5H2upEUH'nTUnUc[ X5TU'M[XU5 M) M4 U$)NF)recurse)rrnamed_parametersidappenditemsrFsetattr) r params_to_tienparamid_key _param_names param_name _tied_params param_init_fns r param_init_fn_tied_param7ensure_weights_retied..param_init_fn_tied_params$D) ///>HA%yL(bi(//2? v&%2$7$7$9 F* $V,=,36+FL(F6+%: r)rFsplitr` get_submodulerrdnnModule) rr0r _tied_namesnamermodrrrs ` @r ensure_weights_retiedrs%!5t._infer_parameter_dtypes =55jAI%,E?$C! !:!g{?P?PTYTgTg?g    . .7M%OOM$B)@)@)BMQQ =0:0A0A#q0I -O++O._cast_and_contiguouss.  YYUY+F &&(F rz Parameter 'zZ' found in sharded model state dict but missing from full state dict. Full state dict has z keys, sharded has z keys.)srcgroupcpu)rrT)assign)torch.distributedrYtorch.distributed.tensorrrr/r}rKeyErrorr device_meshdetachr device_type isinstanceto_local broadcastrWORLD placementsrdemptysizerr7)rhr0rr@distrrmeta_sharded_sd sharded_sdrrr sharded_param full_paramrsharded_tensorrr full_tensors r fsdp2_load_full_state_dictr s'%C&&(OJR$""")8)>)>)@ %J(!*.++.w<.8KCP_L`Kaagi!,J'33K#**,// 0G0GHJ*..(002 NN:1DJJ4D4DN E.z H`H`aN+A, (M= 2.-Q^_N!/!2!25!9%3z "3*A8*9)>)>)@ %J'33K++m&8&8&:;CZCZbobubuvK NN;ATZZ5E5EN F.{KIaIabN+A, (M= 2.-Q^_N!/!2!25!9%3z "*A" *T2 LrrmappingcSSKJn 0nSX2'URH(nUSVs/sHoQURPM snUS'M* gs snf![a [ S5ef=f)a Switches the parameters of the optimizer to new ones (sharded parameters in usual case). This function modifies the optimizer in-place. Args: optimizer (`torch.optim.Optimizer`): Optimizer instance which contains the original model parameters mapping (`dict`): Mapping from the original parameter (specified by `data_ptr`) to the sharded parameter Raises: KeyError: If a parameter in the optimizer couldn't be switched to its sharded version. This should never happen and indicates a bug. If we kept the original params instead of raising, the training wouldn't be numerically correct and weights wouldn't get updated. r)r _local_tensorparamszA parameter in the optimizer couldn't be switched to its sharded version. This breaks the training. Please raise an issue on GitHub.N)rr param_groupsdata_ptrr)rr raccessor_mapping param_groupps r !fsdp2_switch_optimizer_parametersr3sy1 / $11KBMhBW$XBWQQZZ%8BW$XK !2$X   S   sA A A A A"cnSSKJn [URRU5n[ USS9SSH}upE[ URS55S:aURSS5upgOSnUnU(aURU5OUnU"U5(dMeU"US S 9nURXu5 M U$) a Applies the activation checkpointing to the model. Args: accelerator (`Accelerator`): The accelerator instance model (`torch.nn.Module`): The model to apply the activation checkpointing to Returns: `torch.nn.Module`: The model with the activation checkpointing applied r)checkpoint_wrapperT) return_fqnsNrrr F)preserve_rng_state) ;torch.distributed.algorithms._checkpoint.checkpoint_wrapperrfsdp2_prepare_auto_wrap_policyrrHrrrrrregister_module) rhr0rauto_wrap_policy_func layer_namelayer parent_name child_name parent_modules r fsdp2_apply_acr$Rs;;;L;L;X;XZ_`:5dSTWUWX z$ % )&0&7&7Q&? #KK#JR@"S [CU 5S 35 URDS:waU (d/nUR-5HwupU R2(dMU RF[HRJ:wdM8URMU 5 U RNRQ[HRJ5U l'My UR<(a,U(a%[RRUS[CU5SUS35 URW5nURX(aU (d[[US S S9n[\R^"URa5VVs0sHunnUU;dMUU_M snn5nURQ[HR("S55n[cUS5(aURe5 [gXa5nUb@[iU5SSH.nU"U5(dM[ UU5(aM%U"U40UD6 M0 [USS5"5n[USS5"5n[USS5n[USS5nUSL=(a UULnU(dUb[ UU5(d U"U40UD6 [kU5nU(aUOUnUU4Vs/sHnUcM[ UU5(aMUPM nnU(aU"U400UESS 0ED6 [ X5(d7URm5VVs0sHunnUS:wdMUU_M nnnU"U40UD6 URX(a,U (d%SSKJ7n [qXU[ URU 5S9 URX(aU (dWRm5Hcun!n"U"RQUR(5n"SU!;a'U!RsSS5un#n$URuU#5n%OU!n$Un%U%RwU$U"S S 9 Me [cUS5(aURe5 U$s snnfs snfs snnf)!a Prepares the model for FSDP2 in-place. Also returns the model to avoid misuse of the original model. Args: accelerator (`Accelerator`): The accelerator instance model (`torch.nn.Module`): The model to prepare Returns: `torch.nn.Module`: Prepared model r) FSDPModuleMixedPrecisionPolicy fully_shardtorch_device_meshN)reshard_after_forwardoffload_policy mp_policymeshrz2.7.0ignored_paramsF Params4bitTzFound z non-floating frozen Params4bit. Excluding from FSDP2 sharding to prevent quant_state corruption.To enable memory-efficient sharding of 4-bit weights, setbnb_4bit_quant_storage to a floating dtype (e.g. bf16).nozFSDP upcast of low precision parameters to fp32 (since mixed_precision != 'no') may affect the precision of model checkpoints. This effects z parameters: z...)rfqnsmeta tie_weightsrget_input_embeddingscgNrrrr %fsdp2_prepare_model..srget_output_embeddingscgrArrrr rBrCs4rweightr5)CPUOffloadPolicy)r@rr ) persistent)<torch.distributed.fsdpr1r2r3rr _orig_modrrHset_auto_wrap_policyrFr5r@mixed_precision_policytupleparallelism_configfsdp_dim_namesrignored_modulesget_parameters_from_modulesrsetr __class__r+ requires_gradr is_complexaddgetr}warningswarnrmixed_precisionrrdfloat32rdatarrbrcr/cpu_ram_efficient_loadingrcopydeepcopy named_buffersrr>rrr/rrGr rrregister_buffer)&rhr0r1r2r3 is_type_fsdp fsdp2_pluginr8 fsdp2_kwargsmodel_has_params4bitincompatible_params4bitrrignoredupcasted_params original_sdnon_persistent_buffer_fqnskvoriginal_non_persistent_buffersrr input_embed output_embed input_weight output_weightis_weights_tiedr. tail_embedmtail root_kwargsrGfqn buffer_tensor parent_fqnlocal_buffer_namer#s& r fsdp2_prepare_modelr{sxUTe05!Mj*&M $$00L%%e, ; 3T :D".!C!C&22!88R d^s* X .X X+X>X? XXc,^^ SSKJnJn TRn[ U[ R 5(a URnXCLa[USS5nUc/n[U5nTRb TRn[5m UH1n[X5nUc[SUS35eT RU5 M3 S[R R"S[$4U 4S jjn U $XBLa*S[R R"S[$4U4S jjn U $g) aPrepares the auto wrap policy based on its type, done to mimic the behaviour of FSDP1 auto wrap policy. Args: fsdp2_plugin (`FullyShardedDataParallelPlugin`): Instance of `FullyShardedDataParallelPlugin` containing the configuration options model (`torch.nn.Module`): The model to wrap Returns: `Callable[[torch.nn.Module], bool] | None`: The auto wrap policy function to be applied to the model or `None` r)size_based_auto_wrap_policytransformer_auto_wrap_policy_no_split_modulesNz+Could not find the transformer layer class z in the model.rr%c>>T(dg[U[T55$)NF)rrM)rtransformer_cls_to_wraps r policy.fsdp2_prepare_auto_wrap_policy..policyMs*fe,C&DE Ercb>[SUR555nUTR:$)Nc3@# UHoR5v M g7frA)numel).0rs r Afsdp2_prepare_auto_wrap_policy..policy..Us#K7J!GGII7Js)sum parametersmin_num_params)rmodule_num_paramsrcs r rrTs. ##Kv7H7H7J#K K $|'B'BB Br)torch.distributed.fsdp.wrapr}r~auto_wrap_policyr functoolspartialfuncrFrtransformer_cls_names_to_wraprRrr|rVrdrrbool) rcr0r}r~fnno_split_modulesr layer_classtransformer_clsrrs ` @r rr*sf  & &B"i''(( WW )"5*=tD  #! (,-=(>%  5 5 A,8,V,V )"%%8K8LO& #N{m[i!jkk # ' ' 8 9  F588?? Ft F M * C588?? Ct C Mrc SSKJn U"S0UD6$)z Returns a `GradScaler` for FSDP2, as the current implementation of `get_grad_scaler` doesn't accept other args. We need this as current `get_grad_scaler` accepts only `distributed_type` as arg, which doesn't differentiate between FSDP1 and FSDP2 r) GradScalerr)torch.amp.grad_scalerr)kwargsrs r get_fsdp2_grad_scalerr]s 1   r named_paramscUR5VVs0sHupURSS5U_M nnnUR5VVs0sH0upURS5(aURSS5OUU_M2 nnnUR5VVs0sHupURSS5U_M nnnU$s snnfs snnfs snnf)aRemoves parameter name modifiers in order to map them back to their original names. See huggingface/accelerate#3554 for more context. Args: named_params (`dict`): The named parameters dictionary to canonicalize. Returns: `dict`: The canonicalized named parameters dictionary z._checkpoint_wrapped_moduler(z _orig_mod.z ._orig_mod)rreplace startswith)rrkrls r fsdp2_canonicalize_namesrhsQ]PbPbPdePdAII;R@!CPdLeXdXjXjXlXlPTPQq||L'A'A ,#q!KXl@L?Q?Q?ST?StqAIIlB/2?SLT  fUsB: 7CCmodulescUc [5$/n[U[5(am[R"U5n/nUR 5H?upgUR U5(dMURU5 URU5 MA UnUH+nUR[UR555 M- [U5$)zConverts modules to parameters where modules can be a string or list of torch.nn.Module Args: modules (`Union[Iterable[torch.nn.Module], str]`): List of modules Returns: `set[torch.nn.Parameter]`: List of parameters ) rRrstrrecompile named_modules fullmatchrrextendrr)rr0rrregmapped_modulesrrs r rQrQ{su J'3jj!!//1LD}}T"" &!%%f-2!$v00234 z?r)FN)rF)r)T)TF)F)Er^rrrrrX collectionsrcollections.abcr contextlibrpathlibrtypingrrrdloggingr constantsr r rr dataclassesrmodelingrrotherrrrversionsrr+rbr!r$r3r8rIrrrrrrrrrrrrrdictr r Optimizerrr$r/r{rrrrR ParameterrQrrr rs%  #$"" WW3?JJ& H :; " 1"76tEP-@`6Xrccgk:kp44&)4?C4cg4n+$+$+$\]588??]T]`d]@ 1F1F QU >uxxBEHHOO$0F$bEHHOObbJ00HV[V^V^VeVeUfhlUlLmptLt0f 4D& 8EHHOO,c1 2  r