9jTxSSKrSSKrSSKrSSKrSSKJr SSKJr SSKJrJ r SSK J r SSK J r SSKrSSKrSSKJr SSKJr S S KJr S S KJr S S KJr S SKJr S SKJr S SK J!r!J"r"J#r#J$r$J%r% S SK&J'r' S SK(J)r) S SK*J+r+ \"\,5r-\$"5(a SSK.J/s J0r1 S\RdRfS\44Sjr5S\RdRfS\44Sjr6S\RdRfS\44Sjr7S\RdRfS\44Sjr8S\RdRfS\RdRf4Sjr9S\RdRf4Sjr:S\RdRfS\RdRf4Sjr;S\RdRfS\44Sjr<S7S\4S \4S!\44S"jjr=S#r>S$\?4S%jr@S8S&\4S'\44S(jjrA\""S)5(a \RO \R^rC\CRR\R\\R/rH\""S*5(a%\HR\RR5 S9S+jrLS,rMS-rNS9S.\ \OS\44S/jjrPS\O4S0jrQS1rRS2rSS3\T4S4jrUS:S\RdRfS5\4S\V\RdRf4S6jjrWg);N)encode) OrderedDict)partialreduce) MethodType)Optional)Version) save_file)write_basic_config) get_logger) PartialState)FSDP_PYTORCH_VERSION)DistributedType)is_deepspeed_availableis_numpy_availableis_torch_distributed_availableis_torch_xla_availableis_weights_only_available)id_tensor_storage) convert_model)is_torch_versionmodulereturnc[[S5(dg[U[RRR 5$)z< Check whether the module was compiled with torch.compile() _dynamoF)hasattrtorch isinstancer eval_frameOptimizedModulers P/home/wildlama/miniconda3/lib/python3.13/site-packages/accelerate/utils/other.pyis_compiled_moduler%6s1 5) $ $ femm66FF GGc[[S5(dgUR(aLUR5H8n[ U[R R R5(dM8 g g)zT Check whether the module has submodules that were compiled with `torch.compile()`. rFT)rr_modulesmodulesr rr!r"r submodules r$has_compiled_regionsr,@sQ 5) $ $ )I)U]]%=%=%M%MNN* r&c^[T[RR5=(a* [ T5S:=(a [ U4SjT55$)z Check whether the module is a repeated block, i.e. `torch.nn.ModuleList` with all children of the same class. This is useful to determine whether we should apply regional compilation to the module. rc3V># UHn[UTSR5v M g7f)rN)r __class__).0mrs r$ %is_repeated_blocks..Xs%CFq 1fQi1122Fs&))r rnn ModuleListlenallr#s`r$is_repeated_blocksr8OsC 6588../ D K!O D CFC Cr&cxUR(a)UR5Hn[U5(dM g g)z Check whether the module has repeated blocks, i.e. `torch.nn.ModuleList` with all children of the same class, at any level of the module hierarchy. This is useful to determine whether we should apply regional compilation to the module. TF)r(r)r8r*s r$has_repeated_blocksr:\s/ )I!),,* r&c ^S[RRS[RR4U4SjjmT"U40UD6nSUR;aXRS'U$)a Performs regional compilation where we target repeated blocks of the same class and compile them sequentially to hit the compiler's cache. For example, in `GPT2LMHeadModel`, the repeated block/class is `GPT2Block`, and can be accessed as `model.transformer.h[0]`. The rest of the model (e.g. model.lm_head) is compiled separately. This allows us to speed up the compilation overhead / cold start of models like LLMs and Transformers in general. See https://pytorch.org/tutorials/recipes/regional_compilation.html for more details. Args: module (`torch.nn.Module`): The model to compile. **compile_kwargs: Additional keyword arguments to pass to `torch.compile()`. Returns: `torch.nn.Module`: A new instance of the model with some compiled regions. Example: ```python >>> from accelerate.utils import compile_regions >>> from transformers import AutoModelForCausalLM >>> model = AutoModelForCausalLM.from_pretrained("gpt2") >>> compiled_model = compile_regions(model, mode="reduce-overhead") >>> compiled_model.transformer.h[0] OptimizedModule( (_orig_mod): GPT2Block( (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D(nf=2304, nx=768) (c_proj): Conv1D(nf=768, nx=768) (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D(nf=3072, nx=768) (c_proj): Conv1D(nf=768, nx=3072) (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) ) ``` rrc >[U5(aO[RR5nUH)nUR [R "U40UD65 M+ U$[ U5(aURRUR5nURRUR5 0Ul UR5HupCURUT"U40UD65 M U$[R "U40UD6nU$N)r8rr4r5appendcompiler:r/__new____dict__updater(named_children add_module)rcompile_kwargs new_moduler+name_compile_regionss r$rH)compile_regions.._compile_regionss f % %,,.J# !!%-- "L^"LM$! ( ())11&2B2BCJ    & &v 7"$J #)#8#8#:%%d,~ >' ((r&c [U5(aUHnUR"S0UD6 M U$[U5(a%UR5Hn[ U40UD6 M U$UR"S0UD6 U$)a Like `compile_regions`, but uses the in-place `module.compile()` instead of `torch.compile(module)`. Needed for the FSDP2 prepare path: `torch.compile(module)` returns an `OptimizedModule` whose `__call__` bypasses `nn.Module._call_impl`, so forward/pre hooks added later by `fully_shard` never fire and per-layer all-gather/reshard is lost. The in-place `module.compile()` keeps `_call_impl` (and its runtime hook check) on the call path, so FSDP hooks installed afterwards still run. Args: module (`torch.nn.Module`): The model to compile. **compile_kwargs: Additional keyword arguments to pass to `module.compile()`. rN)r8r?r:rOcompile_regions_fsdp2rQs r$rTrTsy&!!I    / /  M V $ $__&E !% :> :' M (( Mr&modelc^[SS5(aSSKJm OSSKJm [ U4SjUR 555$)z Check if the model has DTensor parameters. Args: model (`torch.nn.Module`): The model to check. Returns: `bool`: Whether the model has DTensor parameters. >=z2.5.0r)DTensorc3<># UHn[UT5v M g7fr=)r )r0prXs r$r2$model_has_dtensor..sB/A!z!W%%/As)rtorch.distributed.tensorrXtorch.distributed._tensorany parameters)rUrXs @r$model_has_dtensorr`s4g&&4 6 Bu/?/?/AB BBr&keep_fp32_wrapperkeep_torch_compile recursivec^ [RRR[RR4n[ U5n[ U5nSnU(aUnURnO(U(a!SUR;aUnURSn[5(a SSK J n XH4- n[S[5(a[5(a SSKJn XI4- n[#X5(aUR$n[#X5(aMU(aU 4Sjm T "U5nU(dUR&n URR)SS5n U bF[+U S 5(a%U R,n X:XaO[+U S 5(aM%[/X5Ul[1US S 5(a [3US S 9 U(a+Ub(U(a XlUnU$U(aXRS'UnU$) a_ Extract a model from its distributed containers. Args: model (`torch.nn.Module`): The model to extract. keep_fp32_wrapper (`bool`, *optional*): Whether to remove mixed precision hooks from the model. keep_torch_compile (`bool`, *optional*): Whether to unwrap compiled model. recursive (`bool`, *optional*, defaults to `False`): Whether to recursively extract all cases of `module.module` from `model` as well as unwrap child sublayers recursively, not just the top-level distributed containers. Returns: `torch.nn.Module`: The extracted model. NrJr)DeepSpeedEnginerW)FullyShardedDataParallelc>[US5(aT"UR5nOUnUR5Hup#[XT"U55 M U$)Nr)rrrCsetattr)runwrapped_modulerGrR_recursive_unwraps r$rj6extract_model_from_parallel.._recursive_unwrap*sTvx((#4V]]#C #) />>@ (0A%0HI A# #r&_original_forward __wrapped__ _converted_to_transformer_engineF)to_transformer_engine)rr4parallelDistributedDataParallel DataParallelr%r,rJrAr deepspeedrerrr2torch.distributed.fsdp.fully_sharded_data_parallelrfr rforwardpoprrmrgetattrr) rUrarbrcoptions is_compiled has_compiledcompiled_modelreFSDPruoriginal_forwardrjs @r$extract_model_from_parallelr~s(xx  88%((:O:OPG$U+K'.LN  %.. ("NNN;/E-%%2338V8X8Xg7 U $ $  U $ $ $"%( -- >>--.A4H  ''=11!--.'=11'w6EM 5 Make sure all processes will reach this instruction otherwise one of your processes will hang forever. N)rwait_for_everyonerNr&r$rrPsN$$&r& state_dictc [R"[5nUR5H9up#[ U[ 5(aMU[ U5RU5 M; UR5VVs0sHupE[U5S:dMXE_M nnn[5nUR5H>nUVs/sH o"U;dM UPM nnURUSS5 USSHnX M M@ [U5S:a[RSUS35 UR5V V s0sH7upU [ U [R5(aU R!5OU _M9 nn n U$s snnfs snfs sn n f)z Cleans the state dictionary from a model and removes tensor aliasing if present. Args: state_dict (`dict`): The state dictionary from a model rNrzRemoved shared tensor zk while saving. This should be OK, but check by verifying that you don't receive any warning while reloading) collections defaultdictlistitemsr strrr>r6setvaluesrBloggerwarningrTensor contiguous) rptrsrGtensorptrnames shared_ptrs warn_names found_nameskvs r$ clean_state_dict_for_safetensorsr]sa  " "4 (D"((* &#&& "6* + 2 24 8+ 15 O *#E Q:3: KOJ##% ).D1Ct D+ab/*OD $& :$ZL1\ ] WaVfVfVhiVhda!z!U\\'B'BQ\\^IVhJi #PEjs;E7E7> E= E=4>Fsave_on_each_nodesafe_serializationc[5R[R:Xa[R "U5nU(a1[ [SS0S9n[U[5(a [U5nO[Rn[5R(aU(d U"X5 g[5R(aU(a U"X5 ggg)a Save the data to disk. Use in place of `torch.save()`. Args: obj: The data to save f: The file (or file-like object) to use to save the data save_on_each_node (`bool`, *optional*, defaults to `False`): Whether to only save on the global main process safe_serialization (`bool`, *optional*, defaults to `False`): Whether to save `obj` using `safetensors` or the traditional PyTorch way (that uses `pickle`). formatpt)metadataN)rdistributed_typerXLAxm_maybe_convert_to_cpursafe_save_filer rrrsaveis_main_processis_local_main_process)objfrr save_funcs r$rrs$~&&/*=*==&&s+Nh5EF c; ' '237CJJ ~%%.?#  - -2C#3D -r&z2.0.0z1.25.0c p[5(aM[RR5nSU;aSUS'[RR [ 5 OUR SS5 [R"U4SU0UD6n[5(aD[RR5 W(a[RR U5 U$![5(aF[RR5 W(a [RR U5 fff=f)a Compatible drop-in replacement of `torch.load()` which allows for `weights_only` to be used if `torch` version is 2.4.0 or higher. Otherwise will ignore the kwarg. Will also add (and then remove) an exception for numpy arrays Args: f: The file (or file-like object) to use to load the data map_location: a function, `torch.device`, string or a dict specifying how to remap storage locations **kwargs: Additional keyword arguments to pass to `torch.load()`. weights_onlyTN map_location) rr serializationget_safe_globalsadd_safe_globalsTORCH_SAFE_GLOBALSrvloadclear_safe_globals)rrkwargsold_safe_globals loaded_objs r$rrs G $ & &$22CCE V+)-~&    0 01C D JJ~t ,ZZG GG $ & &    2 2 4##445EF  % & &    2 2 4##445EF  's BCAD5c[US5(d[US5(d [USU5n[US5(a UR$[US5(a UR$[ U5$)z Gets a pretty name from `obj`. __qualname____name__r/)rrwrrr)rs r$get_pretty_namersd 3 ' 'Z0H0Hc;,sN##sJ|| s8Or&cUR5H=up#[U[5(aURU05n[ X45 M9X1U'M? U$)z Recursively merges two dictionaries. Args: source (`dict`): The dictionary to merge into `destination`. destination (`dict`): The dictionary to merge `source` into. )rr dict setdefault merge_dicts)source destinationkeyvaluenodes r$rrsMlln  eT " "))#r2D  $$  % r&portcUcSn[R"[R[R5nURSU45S:HsSSS5 $!,(df  g=f)z Checks if a port is in use on `localhost`. Useful for checking if multiple `accelerate launch` commands have been run and need to see if the port is already in use. Ni> D [[F XX0$,,?NAKw'+..&wi/[\g[his s  sd3 /r&attrcDSn[X /URS5-5$)z Recursive `getattr`. Args: obj: A class instance holding the attribute. attr (`str`): The attribute that is to be retrieved, e.g. 'attribute1.attribute2'. c[X5$r=)rw)rrs r$_getattr#recursive_getattr.._getattr0s s!!r&.)rr)rrrs r$recursive_getattrr%s#" (EDJJsO3 44r& return_fqnscU(dUOSU4nU/n/nU(aUR5nU(aUupeUR5Hmupx[U[RR 5(dM0U(a%W(aUS-U-OUn UR X45 M\UR U5 Mo U(aUR WU45 OUR U5 U(aMUSSS2$)a)Traverse the model in bottom-up order and return the children modules in that order. Args: model (`torch.nn.Module`): the model to get the children of Returns: `list[torch.nn.Module]`: a list of children modules of `model` in bottom-up order. The last element is the `model` itself. rrN)rvrCr rr4rKr>) rUrtopstackordered_modulescurrent_modulecurrent_module_namerGr child_names r$get_module_children_bottom_upr6s#%U C EEO  2@ / (779JD$00EX!4s!:T!A^bJLL*!34LL& :   " "$7#H I  " "> 2 % 4R4  r&)TTF)FFr=)F)Xrrrrcodecsrr functoolsrrtypesrtypingrnumpynprpackaging.versionr safetensors.torchr rcommands.config.defaultr loggingr stater constantsr dataclassesrimportsrrrrrmodelingrtransformer_enginerversionsrrrtorch_xla.core.xla_modelcore xla_modelrr4rKboolr%r,r8r:rLrPrTr`r~rrrr_corenp_core multiarray _reconstructndarraydtyperr>dtypes UInt32DTyperrrintrrrrrrrrrNr&r$rs #% %98 +((-& H ))HuxxH4H  T  uxx 4   D EEHHOOE%((//EP)ehhoo)0%((//4CUXX__CC*_dU"U?CUW[Up '  FDdH)11"((rww ##JJ HH hbii334@ $6#6$6 "s ""4$55"!!t!X\]b]e]e]l]lXm!r&