+j8 d dl Z d dlmZ d dlmZ d dlZd dlZd dlm Z ddl
m Z m Z m
Z
mZmZmZmZmZmZmZmZmZmZmZmZmZmZm Z m Z m Z m Z edd rd d l m!Z! nd d l"m!Z! e r d dl#m$c m%Z& dd
l'm(Z( dd l)m*Z* e(e+ Z, d de-de.e/ de.de.de.de0de0dee! de1de1fdZ2 d dZ3d de0de1fdZ4d de0fd Z5dS )! N)Path)Optional)
load_model )
MODEL_NAMEOPTIMIZER_NAMERNG_STATE_NAMESAFE_MODEL_NAMESAFE_WEIGHTS_NAME SAMPLER_NAME SCALER_NAMESCHEDULER_NAME WEIGHTS_NAMEget_pretty_nameis_cuda_availableis_hpu_availableis_mlu_availableis_musa_availableis_neuron_availableis_sdaa_availableis_torch_versionis_torch_xla_availableis_xpu_availableloadsavez>=z2.4.0)
GradScaler)
get_logger) PartialStateFT
output_dir model_states
optimizers
schedulers dataloaders
process_indexstepscalersave_on_each_nodesafe_serializationc
t | } t | D ]z\ }
} | st nt } |
dk r| dd|
d } | | }
t
| |
|| t d|
{t | D ]{\ }
}| } |
dk r
t dn t d|
d}| | }t
| ||d t d| |t | D ]{\ }
}| } |
dk r
t dn t d|
d}| | }t
| ||d t d | |t | D ] \ }
}|
dk r
t dn t d|
d}| | }d
d l
m}m} t! |j | r7| }t! || rt
|||d t' |d d rL|
dk rd
nd|
d}| | }| } t) j | | t d|
d| |`| } | t* } t) j | | t d| i } t, d| d} || d< t/ j | d< t2 j | d< t) j | d< t9 r!t( j | d< t? r"t( j | d< n_tC r"t( j" | d< n/tG r!t( j$ | d< tK r!t( j& | d < tO r!t( j( | d < tS r!t( j* | d < tW rtY j | d < | | } t) j | | t d | | S )!aw
Saves the current states of the models, optimizers, scaler, and RNG generators to a given directory.
If `safe_serialization` is `True`, models will be saved with `safetensors` while the rest are saved using native
`pickle`.
Args:
output_dir (`str` or `os.PathLike`):
The name of the folder to save all relevant weights and states.
model_states (`List[torch.nn.Module]`):
A list of model states
optimizers (`List[torch.optim.Optimizer]`):
A list of optimizer instances
schedulers (`List[torch.optim.lr_scheduler._LRScheduler]`):
A list of learning rate schedulers
dataloaders (`List[torch.utils.data.DataLoader]`):
A list of dataloader instances to save their sampler states
process_index (`int`):
The current process index in the Accelerator state
step (`int`):
The current step in the internal step tracker
scaler (`torch.amp.GradScaler`, *optional*):
An optional gradient scaler instance to save;
save_on_each_node (`bool`, *optional*):
Whether to save on every node, or only the main node.
safe_serialization (`bool`, *optional*, defaults to `True`):
Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).
r ._)r' r( zModel weights saved in .binFzOptimizer state saved in zScheduler state saved in r IterableDatasetShardSeedableRandomSampleruse_stateful_dataloaderdl_state_dict.bindl_state_dict_z Sampler state for dataloader z
saved in Nz Gradient scaler state saved in .pklr% random_statenumpy_random_seedtorch_manual_seedtorch_xpu_manual_seedtorch_mlu_manual_seedtorch_sdaa_manual_seedtorch_musa_manual_seedtorch_hpu_manual_seedtorch_neuron_manual_seedtorch_cuda_manual_seedxm_seedzRandom states saved in )-r enumerater r replacejoinpathr loggerinfo
state_dictr r r data_loaderr. r/
isinstancedataset get_samplergetattrtorchr
r randomgetstatenp get_state
get_rng_stater xpuget_rng_state_allr mlur sdaar musar hpur neuronr cudar xm)!r r r! r" r# r$ r% r& r' r( istate weights_nameoutput_model_fileoptoptimizer_nameoutput_optimizer_file schedulerscheduler_nameoutput_scheduler_file
dataloader sampler_nameoutput_sampler_filer. r/ samplerdataloader_state_dict_name!output_dataloader_state_dict_filerD output_scaler_filestates states_nameoutput_states_files! [/home/wildlama/comfy/ComfyUI/.venv/lib/python3.11/site-packages/accelerate/checkpointing.pysave_accelerator_statern ? s X j!!J l++ C C5+=T||CT q55'//XXXX>>L & / / = = U%9J_qrrrr A.?AABBBBJ'' I I3 45FFN0000>@[@[TU@[@[@[ * 3 3N C C U)=Nchiiii G0EGGHHHH!*-- I I 9$$&&45FFN0000>@[@[TU@[@[@[ * 3 3N C C U)=Nchiiii G0EGGHHHH";// X X
:01Q,,,,,| ? ? r ,,..G'#899
rW 1EVkpqqqq : 8% @ @ F@AQ)<)"$)"5"5"7"7F "'"5"7"7F H*/)*E*E*G*G&' J*/)*E*E*G*G&'' J+0:+G+G+I+I'(( J+0:+G+G+I+I'( H*/)*E*E*G*G&' N-2\-K-K-M-M)* J+0:+G+G+I+I'( / , . .y#,,[99 Jv)***
KK>*<>>??? c . t }
|dvrt d |d}n|dk rt j }|i }t | } t | D ]\ } } | dk rd| nd}
| t |
d }| r t | |fd
t | i| c| t |
d }t || } | j
|fi | t d
t | D ]a\ } }| dk r
t d n t d| d }| | }t |fd|i|}||
| bt d t | D ]Y\ } }| dk r
t" d n t" d| d }| | }t |fi |}|
| Zt d t | D ]\ } }| dk r
t$ d n t$ d| d }| | }ddlm}m} t- |j | rF| } t- | | r"| t | } t5 |dd rY| dk rdnd| d } | | } | r"t | fi |}|
| t d |X| t6 } t | }!|
|! t d t | t8 d| d }"d|"v r |"d |
d< t; j |"d t> j |"d tC j" |"d tG r%tB j$ % |"d tM r&tB j' % |"d ntQ r&tB j) % |"d ntU r&tB j+ % |"d! ntY r&tB j- % |"d" nYt] r&tB j/ % |"d# n%tB j0 % |"d$ tc rte j" |"d% t d& n*# tf $ r t d' Y nw xY w|
S )(aO
Loads states of the models, optimizers, scaler, and RNG generators from a given directory.
Args:
input_dir (`str` or `os.PathLike`):
The name of the folder to load all relevant weights and states.
models (`List[torch.nn.Module]`):
A list of model instances
optimizers (`List[torch.optim.Optimizer]`):
A list of optimizer instances
schedulers (`List[torch.optim.lr_scheduler._LRScheduler]`):
A list of learning rate schedulers
dataloaders (`List[torch.utils.data.DataLoader]`):
A list of dataloader instances used in your program
process_index (`int`):
The current process index in the Accelerator state
scaler (`torch.amp.GradScaler`, *optional*):
An optional *GradScaler* instance to load
map_location (`str`, *optional*):
What device to load the optimizer state onto. Should be one of either "cpu" or "on_device".
load_kwargs (`dict`, *optional*):
Additional arguments that can be passed to the `load` function.
load_model_func_kwargs (`dict`, *optional*):
Additional arguments that can be passed to the model's `load_state_dict` method.
Returns:
`dict`: Contains the `Accelerator` attributes to override while loading the state.
)Ncpu on_devicezaUnsupported optimizer map location passed, please choose one of `None`, `'cpu'`, or `'on_device'`Nrq rr r r+ z .safetensorsdevicer, ) map_locationz%All model weights loaded successfullyru z(All optimizer states loaded successfullyz(All scheduler states loaded successfullyr r- r0 Fr1 r2 z1All dataloader sampler states loaded successfullyz$GradScaler state loaded successfullyr3 r% r4 r5 r6 r7 r8 r9 r: r; r< r= r> z%All random states loaded successfullyz Could not load random states)4dict TypeErrorr rt r r? rA r
existsr strr r load_state_dictrB rC r r r rE r. r/ rF rG rH set_samplerrI r
r rK setstaterM set_staterJ
set_rng_stater rP set_rng_state_allr rR r rS r rT r rU r rV rW r rX Exception)# input_dirmodelsr! r" r# r$ r& ru load_kwargsload_model_func_kwargsoverride_attributesrY modelendinginput_model_filerD r] r^ input_optimizer_fileoptimizer_stater` ra input_scheduler_filescheduler_staterc rd input_sampler_filer. r/ rf rg input_dataloader_state_dict_fileinput_scaler_file scaler_staterj s# rm load_accelerator_stater s R &&555 o
$ $#~~, YI f%% H H5 EEQr$-- .V&.V.V.VWW " " $ $ H u . c cs<7H7H cLb c c c c ) 1 1Z2M2M2M2M N N .\JJJJ !E !* G G0F G G G G
KK7888 J'' 7 7345FFN0000>@[@[TU@[@[@[ ( 1 1. A A 3^^,^R]^^1
%%o6666
KK:;;; "*-- 3 3 945FFN0000>@[@[TU@[@[@[ ( 1 1. A A 3CC{CC!!/2222
KK:;;;";// 7 7
:01Q,,,,,| ? ? K ,,..G'#899
K$006H1I1IJJ : 8% @ @ 7@AQ)<)r s
( ( ( ( ( ( 2 D'"" *$$$$$$$)))))) *)))))))))
H $( # #u uut*u u u
u
u u
Z u u u u u u~ I I I I XO O OD O O O OU U U U U U U Uro