3jOSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKJr SSKJ r SSK J r J r SSK rSSKrSSKJs Jr \"\SS5bSSKJrJr SSKJr SSKJr S S KJr S S KJr S S K J!r! S S K"J#r#J$r$J%r%J&r&J'r'J(r(J)r)J*r* \*"5(a1SSK+r+\+RXRZR]5(aSSK-r-\&"5(aSSK/J0r0 \'"5(aSSK1J2r2 \)"5(aSSK3J4r4 \("5(aSSK5r5S\64Sjr7Sr8SSSSS.S\RrS\RrS\:S\:S\RrS-S\6S\;\Rr\Rr\Rr44SjjrSNS"\S#\!S$\RrS%\RrS&\RrS'\RrS(\RrS)\:S\;\Rr\Rr44S*jjr?S"\S\@\=\Rr44S+jrA\R4S,\R"R\D\R"R-4S-jjrES.\@\=\Rr4S/\=S0\R"R4S1jrFS2\@\=\R"R4S\@\=\ 44S3jrGSOS4\=S5\6S6\:S7\:S8\:S9\R\=-S:\RS-4S;jjrJSPS4\=4S<jjrKS=rL\S>S?.S@\R"R\-S9\=\R-SA\M4SBjj5rNSCrOSDrPS\@4SEjrQS\@4SFjrRSQS,\R"RS9\=\R-SA\MSG\MSH\MSI\@\=\ 4S-SJ\S\T\R"RS-S\4SKjjrU"SLSM5rVg)RN)contextmanager)partial)AnyIterable distributed) CPUOffloadShardingStrategy)FullyShardedDataParallel)transformer_auto_wrap_policy)UNet2DConditionModel)DiffusionPipeline)SchedulerMixin)convert_state_dict_to_diffusersconvert_state_dict_to_peft deprecateis_accelerate_availableis_peft_availableis_torch_npu_availableis_torchvision_availableis_transformers_available) get_logger)set_peft_model_state_dict) transformsseedc6[R"U5 [RRU5 [R"U5 [ 5(a [R RU5 g[RRU5 g)z Helper function for reproducible behavior to set the seed in `random`, `numpy`, `torch`. Args: seed (`int`): The seed to set. Returns: `None` N) randomrnptorch manual_seedrnpumanual_seed_allcuda)rs R/home/wildlama/miniconda3/lib/python3.13/site-packages/diffusers/training_utils.pyset_seedr%8s[ KKIINN4 d !!$' ""4(cURnUS-nSU- S-nURURS9UR5n[ UR 5[ UR 5:a3USn[ UR 5[ UR 5:aM3UR UR 5nURURS9UR5n[ UR 5[ UR 5:a3USn[ UR 5[ UR 5:aM3UR UR 5nXV- S-nU$)a_ Computes SNR as per https://github.com/TiankaiHang/Min-SNR-Diffusion-Training/blob/521b624bd70c67cee4bdf49225915f5945a872e3/guided_diffusion/gaussian_diffusion.py#L847-L849 for the given timesteps using the provided noise scheduler. Args: noise_scheduler (`NoiseScheduler`): An object containing the noise schedule parameters, specifically `alphas_cumprod`, which is used to compute the SNR values. timesteps (`torch.Tensor`): A tensor of timesteps for which the SNR is computed. Returns: `torch.Tensor`: A tensor containing the computed SNR values for each timestep. ??device).N)alphas_cumprodtor+floatlenshapeexpand)noise_scheduler timestepsr-sqrt_alphas_cumprodsqrt_one_minus_alphas_cumprodalphasigmasnrs r$ compute_snrr:LsF %33N(#-%(>%9c$A!.00 8H8H0I)TZZ\ !'' (3y+? ?1)< !'' (3y+? ?  & &y 7E$A$D$DIL\L\$D$]^g$h$n$n$p! +11 2S5I I(Ei(P% +11 2S5I I ) 0 0 AE =Q C Jr&r)i) lambda_conf temperatureper_token_weights ignore_indexlogitslabelsr<r=r>r?returnchURS:a"[S[UR5S35eURURSS:wa8[S[UR5S[UR5S35eUS::a[S US35eUR U5nUc[ R "XRS 9nOkURUR:wa8[S [UR5S [UR5S35eURURS 9nURSn[R"URSU5URS5S US9RU5n XvRUR5-R5RS5n X-URU R5-R5U - n US:Xa.[ R"SUR U RS9n XU 4$[ R""5 UR%SS9n XmR'U5-nSSS5 UR)5nUS:waU[)U5- n[ R*"USS9n[ R,"UR5R.n[ R0"URU55nUU-RSS9RURS 9*nUWRUR5-R5RS5nUU-URUR5-R5U- n U [)U5U --nUX4$!,(df  GNC=f)a Computes a confidence-aware training loss for token classification-style heads. This loss combines: - `loss_sft`: standard supervised cross-entropy on all non-ignored labels. - `loss_conf`: an entropy penalty applied only on tokens that are already predicted correctly. Args: logits (`torch.Tensor`): Logits of shape `(..., vocab_size)`. labels (`torch.Tensor`): Labels of shape `(...)`, matching `logits.shape[:-1]`. Values set to `ignore_index` are excluded from both losses. lambda_conf (`float`, *optional*, defaults to `0.0`): Weight for the confidence term. temperature (`float`, *optional*, defaults to `1.0`): Temperature used for the entropy term only. Lower values sharpen the distribution and change the strength of the confidence gradients. per_token_weights (`torch.Tensor`, *optional*): Optional weights of shape `(...)` to reweight both losses per token (e.g. schedule-aware weights). Tokens with weight `0` contribute nothing. ignore_index (`int`, *optional*, defaults to `-100`): Ignore index for labels. Returns: `Tuple[torch.Tensor, torch.Tensor, torch.Tensor]`: `(loss, loss_sft, loss_conf)`. r,z.`logits` must have at least 2 dims, got shape .Nz:`labels` shape must match `logits.shape[:-1]`, got labels=z logits=rz`temperature` must be > 0, got )dtypez9`per_token_weights` shape must match `labels` shape, got z != none) reductionr?r r;)r+rF)dimr))ndim ValueErrortupler1ner ones_likerFr.F cross_entropyreshape reshape_assum clamp_minzerosr+no_gradargmaxeqr/softmaxfinfotinylog)r@rAr<r=r>r?validweights vocab_size per_token_nll denom_sftloss_sft loss_confpredcorrect scaled_logitsprobseps log_probsentropy denom_conflosss r$compute_confidence_aware_lossrnqsQ<{{QI%PVP\P\J]I^^_`aa ||v||CR((Hv||I\H]]efklrlxlxfyezz{ |  a:;-qIJJ IIl #E //& =  " "fll 2KERcRiRiLjKkkopuv|wCwCqDpEEFG $&&V\\&:b!JOOr:&r!  j 88GMM22779CCAFI'%((=3F3F*GGLLNQZZHcKK6==O 9,, }}}$''&/) LLNMc%k(:: MM-R 0E ++ekk " ' 'C %//#./I !&&2&.11 1EEGGJJw}}55::<FFqIJ7"WZZ %>>CCE RI eK(94 4D  $$# s #N"" N1interpolation_typec0[5(d [S5eUS:Xa[RRnU$US:Xa[RR nU$US:Xa[RR nU$US:Xa[RRnU$US:Xa[RRnU$US:Xa[RRnU$US:Xa[RRnU$[S US 35e) a Maps a string describing an interpolation function to the corresponding torchvision `InterpolationMode` enum. The full list of supported enums is documented at https://pytorch.org/vision/0.9/transforms.html#torchvision.transforms.functional.InterpolationMode. Args: interpolation_type (`str`): A string describing an interpolation method. Currently, `bilinear`, `bicubic`, `box`, `nearest`, `nearest_exact`, `hamming`, and `lanczos` are supported, corresponding to the supported interpolation modes in torchvision. Returns: `torchvision.transforms.InterpolationMode`: an `InterpolationMode` enum used by torchvision's `resize` transform. zhPlease make sure to install `torchvision` to be able to use the `resolve_interpolation_mode()` function.bilinearbicubicboxnearest nearest_exacthamminglanczoszThe given interpolation mode z is not supported. Currently supported interpolation modes are `bilinear`, `bicubic`, `box`, `nearest`, `nearest_exact`, `hamming`, and `lanczos`.) r ImportErrorrInterpolationModeBILINEARBICUBICBOXNEAREST NEAREST_EXACTHAMMINGLANCZOSrL)rointerpolation_modes r$resolve_interpolation_moders> $ % % v  Z''99BB& % y ('99AA" ! u $'99==  y ('99AA   .'99GG  y ('99AA  y ('99AA  +,>+?@m n  r&unetr3r4noise noisy_latentstargetencoder_hidden_statesdream_detail_preservationc\URRUR5USSS4nSU- S-n X-n Sn [R"5 U"XBU5R n SSS5 SupUR RS:XaLU nX>- R5nURU 5 URX-5n URU5n X4$UR RS:Xa [S5e[SUR R35e!,(df  N=f) a Implements "DREAM (Diffusion Rectification and Estimation-Adaptive Models)" from https://huggingface.co/papers/2312.00210. DREAM helps align training with sampling to help training be more efficient and accurate at the cost of an extra forward step without gradients. Args: `unet`: The state unet to use to make a prediction. `noise_scheduler`: The noise scheduler used to add noise for the given timestep. `timesteps`: The timesteps for the noise_scheduler to user. `noise`: A tensor of noise in the shape of noisy_latents. `noisy_latents`: Previously noise latents from the training loop. `target`: The ground-truth tensor to predict after eps is removed. `encoder_hidden_states`: Text embeddings from the text model. `dream_detail_preservation`: A float value that indicates detail preservation level. See reference. Returns: `tuple[torch.Tensor, torch.Tensor]`: Adjusted noisy_latents and target. Nr)r()NNepsilon v_predictionz/DREAM has not been implemented for v-predictionzUnknown prediction type ) r-r.r+rrWsampleconfigprediction_typedetachmul_addNotImplementedErrorrL)rr3r4rrrrrr-r6 dream_lambdare_noisy_latents_targetpredicted_noise delta_noises r$ compute_dream_and_update_latentsrs*:%3366y7G7GHTXZ^`dIdeN%(>%9c$A!1KL D M.CDKK +N--:.668 &&**+H+VW**[)  ""    / /> A!"STT3O4J4J4Z4Z3[\]] s D D+c0nUR5H[up#[US5(dM[US5nUcM)UR5nUR 5HupgXqUSU3'M M] U$)z@ Returns: A state dict containing just the LoRA parameters. set_lora_layer lora_layerz.lora.) named_moduleshasattrgetattr state_dictitems)rlora_state_dictnamemodulercurrent_lora_layer_sdlora_layer_matrix_name lora_params r$unet_lora_state_dictr)s O**,  6+ , , 6J%(2(=(=(?%:O:U:U:W6*OYtfF3I2J$KL;X - r&modelc[U[5(dU/nUHCnUR5H,nUR(dMUR U5UlM. ME g)z Casts the training parameters of the model to the specified data type. Args: model: The PyTorch model whose parameters will be cast. dtype: The data type to which the model parameters will be cast. N) isinstancelist parameters requires_gradr.data)rrFmparams r$cast_training_paramsr<sM eT " " \\^E""""XXe_ $r&rprefix text_encodercUR5VVs0sH1up4URU5(dMURUS5U_M3 nnn[[ U55n[ X%SS9 gs snnf)a, Sets the `lora_state_dict` into `text_encoder` coming from `transformers`. Args: lora_state_dict: The state dictionary to be set. prefix: String identifier to retrieve the portion of the state dict that belongs to `text_encoder`. text_encoder: Where the `lora_state_dict` is to be set. default) adapter_nameN)r startswithreplacerrr)rrrkvtext_encoder_state_dicts r$!_set_state_dict_into_text_encoderrMss3B2G2G2I2I$!Q\\Z`Ma%199VR !A%2I99XYp9qrlR[\ s A.A.modules_to_savec0nUR5H,up#UcM URSR5XS3'M. U$)Nr_lora_adapter_metadata)r peft_configto_dict)r metadatas module_namers r$_collate_lora_metadatar`sPI.446  @F@R@RS\@]@e@e@gI %;< = 7 r&weighting_scheme batch_size logit_mean logit_std mode_scaler+ generatorcvUS:XaB[R"X#U4XVS9n[RRR U5nU$US:XaT[R "U4XVS9nSU- U[R "[RU-S- 5S-S- U--- nU$[R "U4XVS9nU$)z Compute the density for sampling the timesteps when doing SD3 training. Courtesy: This was contributed by Rafie Walker in https://github.com/huggingface/diffusers/pull/8528. SD3 paper reference: https://huggingface.co/papers/2403.03206v1. logit_normal)meanstdsizer+rmode)rr+rr r,) rnormalnn functionalsigmoidrandcosmathpi)rrrrrr+rus r$%compute_density_for_timestep_samplingrhs >) LLjzmTZ p HH   ' ' * H V # JJZM& N EJ%))DGGaK!O"<"AA"E"IJ J H JJZM& N Hr&cUS:XaUS-R5nU$US:Xa)SSU-- SUS---nS[RU-- nU$[R"U5nU$)z Computes loss weighting scheme for SD3 training. Courtesy: This was contributed by Rafie Walker in https://github.com/huggingface/diffusers/pull/8528. SD3 paper reference: https://huggingface.co/papers/2403.03206v1. sigma_sqrtgcosmapr r,)r/rrrrO)rsigmas weightingbots r$compute_loss_weighting_for_sd3rsz<'T\((*  X %!f*nq619},3'  OOF+ r&cV[R"5 [RR 5(a[RR 5 g[R RR 5(a[RR 5 g[5(a[RR 5 g[[S5(aC[RR 5(a[RR 5 ggg)zN Runs garbage collection. Then clears the cache of the available accelerator. xpuN) gccollectrr# is_available empty_cachebackendsmpsr torch_npur!rrrIr&r$ free_memoryrsJJL zz       ( ( * *   ! ! !!#   599#9#9#;#; $< r&T)offloadmodulesrc' # U(a[SU55(+nU(a4UVs/sH&n[UR55RPM( nnO![ U5S:XdeUSR/nUHnUR U5 M Sv U(a'[ UW5HupFUR U5 M ggs snf!U(a'[ UW5HupFUR U5 M ff=f7f)a Context manager that, if offload=True, moves each module to `device` on enter, then moves it back to its original device on exit. Args: device (`str` or `torch.Device`): Device to move the `modules` to. offload (`bool`): Flag to enable offloading. c3B# UHn[U[5v M g7fN)rr).0rs r$ !offload_models..sMW:a):;;Wsr rN)anynextrr+r0r.zip)r+rris_modelroriginal_devicesorig_devs r$offload_modelsrsMWMMM ELMWQ\\^ 4 ; ;W M w<1$ $$ ' 1 12 A DDL "7,<= X >  N "7,<= X > s(*D-C >DC4D0DDc^U(d [S5eUR5RS5n/nUHn[R"SU5nU(d[SUS35e[ UR S55n[ UR S55nUS::dUS::a [S 5eUS -S:wd US -S:wa[R"S US US 35 URXV45 M U(d [S5eU$![an[SUSU35UeSnAff=f)zGParses a string defining buckets into a list of (height, width) tuples.zBucket string cannot be empty.;z^\s*(\d+)\s*,\s*(\d+)\s*$zInvalid bucket format: 'z'. Expected 'height,width'.r r,rz,Bucket dimensions must be positive integers.zBucket dimension (,z.) not divisible by 8. This might cause issues.z Invalid integer in bucket pair 'z': Nz.No valid buckets found in the provided string.) rLstripsplitrematchintgroupwarningswarnappend) buckets_str bucket_pairsparsed_bucketspair_strrheightwidthes r$parse_buckets_stringrs/ 9::$$&,,S1LN 5x@7zA\]^ ^ YQ(F A'E{eqj !OPPzQ%!)q. 26(!E7Bpqr  ! !6/ 2! IJJ   Y?zQCPQWX X Ys'B D  D,D''D,c[S5nSn[U5H%unupg[X-X-- 5nX::dM!UnUnM' U$)z6Finds the closes bucket to the given height and width.infN)r/ enumerateabs) hwbucket_options min_metricbest_bucket_idx bucket_idxbucket_hbucket_wmetrics r$find_nearest_bucketrsSuJO,5n,E( (XQ\AL01  J(O -F r&c UR5VVs0sHSupU[U[R5(a,UR 5R 5R 5OU_MU snn$s snnfr)rrrTensorrcpu contiguous) state_dictsrrs r$_to_cpu_contiguousr$sW_j_p_p_r s_rW[WXA 1ell0K0Kqxxz~~**,QR R_r ss ssAA2c0n[URSS5nUc [S5eURRnUc[R US'U$UR =(d [R US'U$)zL Extract and convert FSDP config from Accelerator into PyTorch FSDP kwargs. fsdp_pluginNzLAccelerate isn't configured to handle FSDP. Please update your installation.sharding_strategy)rstaterLr&r FULL_SHARDr') acceleratorkwargs fsdp_stater&s r$ get_fsdp_kwargs_from_acceleratorr-s F**M4@Jghh##//K&6&A&A"# M'2&C&C&bGWGbGb"# Mr&use_orig_paramslimit_all_gathers fsdp_kwargstransformer_layer_clscT[[5nUcJ[URRR S5nUR SUR35 [[U1S9nUU(a [US9OSUUUS.n U(aU RU5 [U40U D6n U $)u) Wrap a model with FSDP using common defaults and optional transformer auto-wrapping. Args: model: Model to wrap device: Target device (e.g., accelerator.device) offload: Whether to enable CPU parameter offloading use_orig_params: Whether to use original parameters limit_all_gathers: Whether to limit all gathers fsdp_kwargs: FSDP arguments (sharding_strategy, etc.) — usually from Accelerate config transformer_layer_cls: Classes for auto-wrapping (if not using policy from fsdp_kwargs) Returns: FSDP-wrapped model Nrz8transformer_layer_cls is not provided, auto-inferred as )r1)offload_params) device_id cpu_offloadr.r/auto_wrap_policy) r__name__typerlanguage_modellayersinforr rupdateFSDP) rr+rr.r/r0r1loggerr6r fsdp_models r$wrap_with_fsdpr@s2 !F$ $U[[%?%?%F%Fq%I J NOdOmOmNnop;TiSjk=Dz9$*., F k"e&v&J r&c<\rSrSrSrSS\\RRS\ S\ S\ S\ S \ \ -S \ \ -S \ S \ S-S \ \\ 4S-4Sjjr\SS Sjj5rSrS\ S\ 4Sjr\R("5S\\RR4Sj5rS\\RRSS4SjrS!SjrS"S!SjjrS\ 4SjrS\\RRSS4SjrS\\RRSS4SjrS\ SS4SjrSrg)#EMAModeli;z. Exponential Moving Average of models weights Nrdecay min_decayupdate_after_stepuse_ema_warmup inv_gammapowerforeach model_cls model_configc  [U[RR5(a Sn [ SSU SS9 UR 5nSnU R SS5bS n [ SSU SS9 U SnU R S S5bS n [ S SU SS9 U S n[U5nUV s/sH oR5R5PM" sn Ul U R S S5b S n [ S SU SS9 URU S S9 SUl X l X0lX@lXPlX`lXplSUlSUlXlXlXlgs sn f)a Args: parameters (Iterable[torch.nn.Parameter]): The parameters to track. decay (float): The decay factor for the exponential moving average. min_decay (float): The minimum decay factor for the exponential moving average. update_after_step (int): The number of steps to wait before starting to update the EMA weights. use_ema_warmup (bool): Whether to use EMA warmup. inv_gamma (float): Inverse multiplicative factor of EMA warmup. Default: 1. Only used if `use_ema_warmup` is True. power (float): Exponential factor of EMA warmup. Default: 2/3. Only used if `use_ema_warmup` is True. foreach (bool): Use torch._foreach functions for updating shadow parameters. Should be faster. device (str | torch.device | None): The device to store the EMA weights on. If None, the EMA weights will be stored on CPU. @crowsonkb's notes on EMA Warmup: If gamma=1 and power=1, implements a simple average. gamma=1, power=2/3 are good values for models you plan to train for a million or more steps (reaches decay factor 0.999 at 31.6K steps, 0.9999 at 1M steps), gamma=1, power=3/4 for models you plan to train for less (reaches decay factor 0.999 at 10K steps, 0.9999 at 215.4k steps). zzPassing a `torch.nn.Module` to `ExponentialMovingAverage` is deprecated. Please pass the parameters of the module instead.z9passing a `torch.nn.Module` to `ExponentialMovingAverage`1.0.0F standard_warnT max_valueNzCThe `max_value` argument is deprecated. Please use `decay` instead. min_valuezGThe `min_value` argument is deprecated. Please use `min_decay` instead.r+z=The `device` argument is deprecated. Please use `to` instead.r*r)rrrModulerrgetrcloner shadow_paramsr.temp_stored_paramsrCrDrErFrGrHoptimization_stepcur_decay_valuerIrJrK)selfrrCrDrErFrGrHrIrJrKr+deprecation_messageps r$__init__EMAModel.__init__@sgF j%((// 2 2D  K##   $..0J"N ::k4 ( 4"g  k7,?u U;'E ::k4 ( 4"k  k7,?u U{+I*% :DE*Qggi..0*E ::h % 1"a  h)> !ij j    $lm m**4+<+<=__& -   .:. U%%'( d#r&rWc[SXR- S- 5nUS::agUR(a#SSX R- -UR*-- nO SU-SU-- n[ X0R 5n[X0R5nU$)z> Compute the decay factor for the exponential moving average. rr r; )maxrErFrGrHminrCrD)rYrWsteprXs r$ get_decayEMAModel.get_decays1'*@*@@1DE 19   1tnn'<#<$**"LLO 4xBI6Oozz:o~~>r&c T[U[RR5(aSn[ SSUSS9 UR 5n[ U5nU=RS- slURUR5nX0l SU- n[R"5nUR(Ga~[5(aK[RR R#5(a[ R$R'USS9nU UVs/sHofR((dMUPM nn[+UR,U5VVs/sHupUR((dMUPM n nn[/U5[/U5:au[R0"[+UR,U5VVs/sHupUR((aMUPM snnUVs/sHofR((aMUPM snSS 9 [R2"U [R4"X5US 9 SSS5 g[+UR,U5Hup[5(aK[RR R#5(a[ R$R'USS9nU UR((aUR7XHU- -5 OUR9U5 SSS5 M gs snfs snnfs snnfs snf!,(df  g=f!,(df  M=f) NzPassing a `torch.nn.Module` to `ExponentialMovingAverage.step` is deprecated. Please pass the parameters of the module instead.z>passing a `torch.nn.Module` to `ExponentialMovingAverage.step`rMFrNr ) modifier_rankT) non_blocking)r7)rrrrRrrrrWrtrX contextlib nullcontextrIr transformers integrations deepspeedis_deepspeed_zero3_enabledzeroGatheredParametersrrrUr0_foreach_copy_ _foreach_sub_ _foreach_subsub_copy_) rYrrZrCone_minus_decaycontext_managerr params_grads_param s_params_grads r$rs EMAModel.stepso j%((// 2 2D  P##   $..0J*%  !#t556$e)$002 <<<(**|/H/H/R/R/m/m/o/o"+.."C"CJ^b"C"c 2<T*@S@Su* T25d6H6H*2U!2UY^YlYlG2U!{#c*o5((7:4;M;Mz7Zv7Z^Wbgbubu7Zv,6RJ5>Q>QJR%) ##!5#5#5m#QYh!$#&d&8&8*"E,..<3L3L3V3V3q3q3s3s&/nn&G&G]a&G&bO$** _%%HI e, %_ #F#U! wR!,%_sgLK1.K14LK6*K60AL4K< K< LL 4L :0L*:L1L L L' c [U5nUR(a~[R"UVs/sHo"RPM sn[ UR U5VVs/sH*up2URUR5RPM, snn5 g[ UR U5HCup2URRURUR5R5 ME gs snfs snnf)a1 Copy current averaged parameters into given collection of parameters. Args: parameters: Iterable of `torch.nn.Parameter`; the parameters to be updated with the stored moving averages. If `None`, the parameters with which this `ExponentialMovingAverage` was initialized will be used. N) rrIrrrrrUr.r+r)rYrrrs r$rlEMAModel.copy_tos*% <<  )344EHI[I[]gEhiEh>7ELL)..Ehi  #&d&8&8*"E   ELL!9!>!>?#F 5is C9!1C> clURVs/sHoR5PM snUlgs snf)z Move internal buffers of the ExponentialMovingAverage to pinned memory. Useful for non-blocking transfers for offloading EMA params to the host. N)rU pin_memory)rYr[s r$rEMAModel.pin_memory s* 7;6H6HI6Hlln6HIIs1c URVs/sH7nUR5(aURXUS9OURXS9PM9 snUlgs snf)z Move internal buffers of the ExponentialMovingAverage to `device`. Args: device: like `device` argument to `torch.Tensor.to` )r+rFrx)r+rxN)rUis_floating_pointr.)rYr+rFrxr[s r$r. EMAModel.tose''  (""$$ DD,D GV? @(   s>Ac URURURURURUR UR URS.$)z Returns the state of the ExponentialMovingAverage as a dict. This method is used by accelerate during checkpointing to save the ema state dict. rCrDrWrErFrGrHrUr)rYs r$rEMAModel.state_dict$sNZZ!%!7!7!%!7!7"11ZZ!//  r&cUVs/sH.o"R5R5R5PM0 snUlgs snf)z Saves the current parameters for restoring later. Args: parameters: Iterable of `torch.nn.Parameter`. The parameters to be temporarily stored. N)rr!rTrV)rYrrs r$storeEMAModel.store7s6NX"XZE<<>#5#5#7#=#=#?Z"X"Xs5AcURc [S5eUR(aV[R"UVs/sHo"R PM snURVs/sHo3R PM sn5 OD[ URU5H*up2UR RUR 5 M, SUlgs snfs snf)a Restore the parameters stored with the `store` method. Useful to validate the model with EMA parameters without: affecting the original optimization process. Store the parameters before the `copy_to()` method. After validation (or model saving), use this to restore the former parameters. Args: parameters: Iterable of `torch.nn.Parameter`; the parameters to be updated with the stored parameters. If `None`, the parameters with which this `ExponentialMovingAverage` was initialized will be used. NzGThis ExponentialMovingAverage has no `store()`ed weights to `restore()`)rV RuntimeErrorrIrrrrr)rYrrc_params r$restoreEMAModel.restore@s  " " *hi i <<  )344SWSjSj6kSj||Sj6k #&d&=&=z"J   .#K#'56ks C #C rcH[R"U5nURSUR5UlURS:dURS:a [ S5eURSUR 5Ul[ UR [5(d [ S5eURSUR5Ul[ UR[5(d [ S5eURS UR5Ul [ UR[5(d [ S 5eURS UR5Ul [ UR[5(d [ S 5eURS UR5Ul [ UR[[45(d [ S5eURSUR5Ul[ UR[[45(d [ S5eURSS5nUb]X l[ UR[ 5(d [ S5e[#SUR55(d [ S5egg)z Loads the ExponentialMovingAverage state. This method is used by accelerate during checkpointing to save the ema state dict. Args: state_dict (dict): EMA state. Should be an object returned from a call to :meth:`state_dict`. rCr;r)zDecay must be between 0 and 1rDzInvalid min_decayrWzInvalid optimization_steprEzInvalid update_after_steprFzInvalid use_ema_warmuprGzInvalid inv_gammarHz Invalid powerrUNzshadow_params must be a listc3V# UHn[U[R5v M! g7fr)rrr )rr[s r$r+EMAModel.load_state_dict..s O@4@&J   D &Y);); <YY'(588+=+=">'4'2.F$.F4.Fr&rB)r))NNNr!Nr)TTTNN)Wryrrrrrrr functoolsrtypingrrnumpyrrtorch.nn.functionalrrrPrtorch.distributed.fsdprr r r=torch.distributed.fsdp.wrapr modelsr pipelinesr schedulersrutilsrrrrrrrrr{r|r}r~accelerate.loggingrpeftr torchvisionrrrr%r:r r/rMrnrrrrrfloat32rRrrrrr+ Generatorrrrrrrrr$r-setr8r@rBrIr&r$rs0 %   5-&2CGH((&     **EEGG-.&)3)("R-1S% LLS% LLS% S%  S% ||d* S%S% 5<<u|| 34S%l)3)h(+3# 3##3#||3# << 3# << 3# LL 3#!<<3# %3# 5<< %&3#l3S%,,=N8O&PU}}-$uxx2G G-"]#u||+,]69]IN]&Dehhoo1E,F4PSUXPX>!&(,        LL3   % 6S$  nrUXX__/@@#PUP\P\J\gk>8 tttT4 ")-?C/ 88??/ %,, // /  / c3h$& /tEHHOO45</ /fLFLFr&