l0jff.ddlZddlZddlZddlZddlZddlmZmZmZm Z m Z m Z m Z ddl ZddlZddlmZddlmZddlmZmZmZmZmZmZmZer&ddlZejjrddlZerddl m!Z!erddl"m#Z#erddl$Z$d e%fd Z&d Z'd e(fd Z) d*dededej*dej*dej*dej*dej*de+de e ej*e ej*ffdZ,dedee(ej*ffdZ-ej.fde ej/j0e ej/j0ffdZ1dee(ej*fde(dej/j0fdZ2 d+d e(d!e%d"e+d#e+d$e+f d%Z3d,d e(fd&Z4d'Z5Gd(d)Z6dS)-N)AnyDictIterableListOptionalTupleUnion)UNet2DConditionModel)SchedulerMixin)convert_state_dict_to_diffusersconvert_state_dict_to_peft deprecateis_peft_availableis_torch_npu_availableis_torchvision_availableis_transformers_available)set_peft_model_state_dict) transformsseedc0tj|tj|tj|t r!tj|dStj|dS)z Helper function for reproducible behavior to set the seed in `random`, `numpy`, `torch`. Args: seed (`int`): The seed to set. Returns: `None` N) randomrnptorch manual_seedrnpumanual_seed_allcuda)rs [/home/wildlama/miniconda3/envs/lam/lib/python3.11/site-packages/diffusers/training_utils.pyset_seedr (s KINN4 d) !!$''''' ""4(((((c|j}|dz}d|z dz}||j|}t |jt |jkr2|d}t |jt |jk2||j}||j|}t |jt |jkr2|d}t |jt |jk2||j}||z dz}|S)a Computes SNR as per https://github.com/TiankaiHang/Min-SNR-Diffusion-Training/blob/521b624bd70c67cee4bdf49225915f5945a872e3/guided_diffusion/gaussian_diffusion.py#L847-L849 for the given timesteps using the provided noise scheduler. Args: noise_scheduler (`NoiseScheduler`): An object containing the noise schedule parameters, specifically `alphas_cumprod`, which is used to compute the SNR values. timesteps (`torch.Tensor`): A tensor of timesteps for which the SNR is computed. Returns: `torch.Tensor`: A tensor containing the computed SNR values for each timestep. ??device).N)alphas_cumprodtor&floatlenshapeexpand)noise_scheduler timestepsr(sqrt_alphas_cumprodsqrt_one_minus_alphas_cumprodalphasigmasnrs r compute_snrr5<s] %3N(#-%(>%9c$A!.00 8H0II)TZZ\\ !' ( (3y+?+? ? ?1)< !' ( (3y+?+? ? ?  & &y 7 7E$A$D$DIL\$D$]$]^g$h$n$n$p$p! +1 2 2S5I5I I I(Ei(P% +1 2 2S5I5I I I ) 0 0 A AE 5=Q C Jr!interpolation_typectstd|dkrtjj}n|dkrtjj}n|dkrtjj}ns|dkrtjj}n[|dkrtjj}nC|dkrtjj }n+|dkrtjj }ntd |d |S) a Maps a string describing an interpolation function to the corresponding torchvision `InterpolationMode` enum. The full list of supported enums is documented at https://pytorch.org/vision/0.9/transforms.html#torchvision.transforms.functional.InterpolationMode. Args: interpolation_type (`str`): A string describing an interpolation method. Currently, `bilinear`, `bicubic`, `box`, `nearest`, `nearest_exact`, `hamming`, and `lanczos` are supported, corresponding to the supported interpolation modes in torchvision. Returns: `torchvision.transforms.InterpolationMode`: an `InterpolationMode` enum used by torchvision's `resize` transform. zhPlease make sure to install `torchvision` to be able to use the `resolve_interpolation_mode()` function.bilinearbicubicboxnearest nearest_exacthamminglanczoszThe given interpolation mode z is not supported. Currently supported interpolation modes are `bilinear`, `bicubic`, `box`, `nearest`, `nearest_exact`, `hamming`, and `lanczos`.) r ImportErrorrInterpolationModeBILINEARBICUBICBOXNEAREST NEAREST_EXACTHAMMINGLANCZOS ValueError)r6interpolation_modes rresolve_interpolation_moderJas $ % %  v   Z'''9B y ( ('9A u $ $'9= y ( ('9A  . .'9G y ( ('9A y ( ('9A n,> n n n   r!r$unetr.r/noise noisy_latentstargetencoder_hidden_statesdream_detail_preservationreturncL|j|j|dddf}d|z dz} | |z} d} tj5||||j} dddn #1swxYwYd\} } |jjdkr\| }||z }| | | | |z} | |} n;|jjdkrtdtd|jj| | fS) a Implements "DREAM (Diffusion Rectification and Estimation-Adaptive Models)" from http://arxiv.org/abs/2312.00210. DREAM helps align training with sampling to help training be more efficient and accurate at the cost of an extra forward step without gradients. Args: `unet`: The state unet to use to make a prediction. `noise_scheduler`: The noise scheduler used to add noise for the given timestep. `timesteps`: The timesteps for the noise_scheduler to user. `noise`: A tensor of noise in the shape of noisy_latents. `noisy_latents`: Previously noise latents from the training loop. `target`: The ground-truth tensor to predict after eps is removed. `encoder_hidden_states`: Text embeddings from the text model. `dream_detail_preservation`: A float value that indicates detail preservation level. See reference. Returns: `tuple[torch.Tensor, torch.Tensor]`: Adjusted noisy_latents and target. Nr$r#)NNepsilon v_predictionz/DREAM has not been implemented for v-predictionzUnknown prediction type ) r(r)r&rno_gradsampleconfigprediction_typedetachmul_addNotImplementedErrorrH)rKr.r/rLrMrNrOrPr(r1 dream_lambdapred_noisy_latents_targetpredicted_noise delta_noises r compute_dream_and_update_latentsrcs:%366y7GHHTXZ^`dIdeN%(>%9c$A!12KKL D LLtM9.CDDKLLLLLLLLLLLLLLL+NG-::.6688 &&&&**+H;+VWW**[))   /> A A!"STTT\O4J4Z\\]]] 7 ""s A++A/2A/ci}|D]_\}}t|drJt|d}|8|}|D]\}}|||d|<`|S)zL Returns: A state dict containing just the LoRA parameters. set_lora_layer lora_layerNz.lora.) named_moduleshasattrgetattr state_dictitems)rKlora_state_dictnamemodulerfcurrent_lora_layer_sdlora_layer_matrix_name lora_params runet_lora_state_dictrrs O**,,ZZ f 6+ , , Z 66J%(2(=(=(?(?%:O:U:U:W:WZZ6*JOYOt$K$K3I$K$KLL r!modelct|ts|g}|D]:}|D]#}|jr|||_$;dS)z Casts the training parameters of the model to the specified data type. Args: model: The PyTorch model whose parameters will be cast. dtype: The data type to which the model parameters will be cast. N) isinstancelist parameters requires_gradr)data)rsdtypemparams rcast_training_paramsr}so eT " " --\\^^ - -E" -"XXe__  ---r!rlprefix text_encodercfd|D}tt|}t||ddS)aD Sets the `lora_state_dict` into `text_encoder` coming from `transformers`. Args: lora_state_dict: The state dictionary to be set. prefix: String identifier to retrieve the portion of the state dict that belongs to `text_encoder`. text_encoder: Where the `lora_state_dict` is to be set. cpi|]2\}}||d|3S)) startswithreplace).0kvr~s r z5_set_state_dict_into_text_encoder..sR*.!QQ\\Z`MaMa 99VR "Ar!default) adapter_nameN)rkrr r)rlr~rtext_encoder_state_dicts ` r!_set_state_dict_into_text_encoderrso2A2G2G2I2I99XYp9q9qrrl,CR[\\\\\\r!weighting_scheme batch_size logit_mean logit_std mode_scalecd|dkr>tj|||fd}tjj|}nk|dkrNtj|fd}d|z |tjtj|zdz dzdz |zzz }ntj|fd}|S)z Compute the density for sampling the timesteps when doing SD3 training. Courtesy: This was contributed by Rafie Walker in https://github.com/huggingface/diffusers/pull/8528. SD3 paper reference: https://arxiv.org/abs/2403.03206v1. logit_normalcpu)meanstdsizer&mode)rr&r r') rnormalnn functionalsigmoidrandcosmathpi)rrrrrus r%compute_density_for_timestep_samplingrs>)) LjizmTY Z Z Z H  ' ' * * V # # JZM% 8 8 8 EJ%)DGaK!O"<"<"AA"E"IJ J JZM% 8 8 8 Hr!c|dkr|dz}n>|dkr$dd|zz d|dzzz}dtj|zz }ntj|}|S)z Computes loss weighting scheme for SD3 training. Courtesy: This was contributed by Rafie Walker in https://github.com/huggingface/diffusers/pull/8528. SD3 paper reference: https://arxiv.org/abs/2403.03206v1. sigma_sqrtgcosmapr r')r*rrr ones_like)rsigmas weightingbots rcompute_loss_weighting_for_sd3rsr<''T\((** X % %!f*nq619},3' OF++ r!ctjtjr tjdStjjr tjdStr tj dSdS)zV Runs garbage collection. Then clears the cache of the available accelerator. N) gccollectrr is_available empty_cachebackendsmpsr torch_npurr!r free_memoryr"sJLLL z  $        ( ( * *$   ! !$ !!#####$$r!ceZdZdZ d"d eejjd ed ed e d e de ee fde ee fde de e deee ffdZed#d$dZdZde defdZejd eejjfdZd eejjddfdZd%dZd&d%dZdefdZd eejjddfdZd eejjddfdZd eddfd!ZdS)'EMAModelz6 Exponential Moving Average of models weights H.?rFr$UUUUUU?Nrwdecay min_decayupdate_after_stepuse_ema_warmup inv_gammapowerforeach model_cls model_configc  t|tjjr+d} t dd| d|}d}| ddd } t dd| d| d}| d dd } t d d| d| d }t|}d |D|_| d d1d} t d d| d| | d d|_ ||_ ||_ ||_ ||_||_||_d|_d|_||_| |_| |_dS)a{ Args: parameters (Iterable[torch.nn.Parameter]): The parameters to track. decay (float): The decay factor for the exponential moving average. min_decay (float): The minimum decay factor for the exponential moving average. update_after_step (int): The number of steps to wait before starting to update the EMA weights. use_ema_warmup (bool): Whether to use EMA warmup. inv_gamma (float): Inverse multiplicative factor of EMA warmup. Default: 1. Only used if `use_ema_warmup` is True. power (float): Exponential factor of EMA warmup. Default: 2/3. Only used if `use_ema_warmup` is True. foreach (bool): Use torch._foreach functions for updating shadow parameters. Should be faster. device (Optional[Union[str, torch.device]]): The device to store the EMA weights on. If None, the EMA weights will be stored on CPU. @crowsonkb's notes on EMA Warmup: If gamma=1 and power=1, implements a simple average. gamma=1, power=2/3 are good values for models you plan to train for a million or more steps (reaches decay factor 0.999 at 31.6K steps, 0.9999 at 1M steps), gamma=1, power=3/4 for models you plan to train for less (reaches decay factor 0.999 at 10K steps, 0.9999 at 215.4k steps). zzPassing a `torch.nn.Module` to `ExponentialMovingAverage` is deprecated. Please pass the parameters of the module instead.z9passing a `torch.nn.Module` to `ExponentialMovingAverage`1.0.0F standard_warnT max_valueNzCThe `max_value` argument is deprecated. Please use `decay` instead. min_valuezGThe `min_value` argument is deprecated. Please use `min_decay` instead.cZg|](}|)Sr)clonerYrps r z%EMAModel.__init__..ts,EEEQaggii..00EEEr!r&z=The `device` argument is deprecated. Please use `to` instead.r%r)rurrModulerrwgetrv shadow_paramsr)temp_stored_paramsrrrrrroptimization_stepcur_decay_valuerrr) selfrwrrrrrrrrrkwargsdeprecation_messages r__init__zEMAModel.__init__6sF j%(/ 2 2 "D  K##      $..00J"N ::k4 ( ( 4"g  k7,?u U U U U;'E ::k4 ( ( 4"k  k7,?u U U U U{+I*%% EE*EEE ::h % % 1"a  h) !ijj j   $lmm m**4+<==__&& ---  ..:... U%%''((( d#####r!rctd||jz dz }|dkrdS|jrdd||jz z|j zz }n d|zd|zz }t ||j}t||j}|S)zN Compute the decay factor for the exponential moving average. rr r )maxrrrrminrr)rrsteprs r get_decayzEMAModel.get_decays1'$*@@1DEE 1993   71tdn'<#<$*"LLOO 4xBI6Ootz::ot~>>r!ct|tjjr)d}t dd|d|}t |}|xjdz c_||j}||_ d|z }tj }|j r"trDtjjr!t j|d}|5d|D}d t)|j|D}t-|t-|kr>tjd t)|j|Dd |Dd tj|tj|||ddddS#1swxYwYdSt)|j|D]\}} trDtjjr!t j| d}|5| jr|||| z zn|| dddn #1swxYwYdS)NzPassing a `torch.nn.Module` to `ExponentialMovingAverage.step` is deprecated. Please pass the parameters of the module instead.z>passing a `torch.nn.Module` to `ExponentialMovingAverage.step`rFrr ) modifier_rankc g|] }|j | Srrxrr|s rrz!EMAModel.step..s TTT@STuTTTr!c&g|]\}}|j |Srrrs_paramr|s rrz!EMAModel.step..s5!!! .Y^Yl!!!!r!c&g|]\}}|j |Srrrs rrz!EMAModel.step..s%vvv^Webgbuvvvvr!c g|] }|j | Srrrs rrz!EMAModel.step..s RRR5e>QRRRRr!T) non_blocking)r2)rurrrrrwrvrrr contextlib nullcontextrr transformers integrations deepspeedis_deepspeed_zero3_enabledzeroGatheredParametersziprr+_foreach_copy_ _foreach_sub_ _foreach_subrxsub_copy_) rrwrrone_minus_decaycontext_manager params_grad s_params_gradrr|s rrz EMAModel.stepsF j%(/ 2 2 1D  P##      $..00J*%%  !#t566$e)$022 < -(** d|/H/R/m/m/o/o d"+."C"CJ^b"C"c"c   TT*TTT !!25d6H*2U2U!!! {##c*oo55(vvs4;Mz7Z7ZvvvRRJRRR%) #!5#5m[#Q#QYh                  $#&d&8*"E"E - -,..c<3L3V3q3q3s3sc&/n&G&G]a&G&b&bO$--*- _%%HIIII e,,, ---------------  - -s%=B4F>>GG99I>>J J cRt|}|jr>tjd|Ddt |j|DdSt |j|D]<\}}|j||j j=dS)aa Copy current averaged parameters into given collection of parameters. Args: parameters: Iterable of `torch.nn.Parameter`; the parameters to be updated with the stored moving averages. If `None`, the parameters with which this `ExponentialMovingAverage` was initialized will be used. cg|] }|j Srryrs rrz$EMAModel.copy_to..444444r!cRg|]$\}}||jj%Sr)r)r&ryrs rrz$EMAModel.copy_to..s-iii>7EEL)).iiir!N) rvrrrrrryrr)r&)rrwrr|s rrzEMAModel.copy_tos*%% < @  44444iiSI[]gEhEhiii      #&d&8*"E"E @ @   EL!9!9!>???? @ @r!c2d|jD|_dS)z Move internal buffers of the ExponentialMovingAverage to pinned memory. Useful for non-blocking transfers for offloading EMA params to the host. c6g|]}|Sr) pin_memoryrs rrz'EMAModel.pin_memory.. s IIIallnnIIIr!Nrrs rrzEMAModel.pin_memorys$ JId6HIIIr!c@fd|jD|_dS)z Move internal buffers of the ExponentialMovingAverage to `device`. Args: device: like `device` argument to `torch.Tensor.to` cg|]E}|r|n|FS))r&rzr)r&r)is_floating_pointr))rrr&rzrs rrzEMAModel.to..sg   ""$$ @ADDe,D G G GV,??   r!Nr)rr&rzrs ```rr)z EMAModel.to sE      '    r!c h|j|j|j|j|j|j|j|jdS)z Returns the state of the ExponentialMovingAverage as a dict. This method is used by accelerate during checkpointing to save the ema state dict. rrrrrrrrrrs rrjzEMAModel.state_dicts?Z!%!7!%!7"1Z!/   r!c(d|D|_dS)z Saves the current parameters for restoring later. Args: parameters: Iterable of `torch.nn.Parameter`. The parameters to be temporarily stored. c~g|]:}|;Sr)rYrrrs rrz"EMAModel.store..4s8"X"X"XE5<<>>#5#5#7#7#=#=#?#?"X"X"Xr!N)r)rrws rstorezEMAModel.store-s!#Y"XZ"X"X"Xr!c |jtd|jr/tjd|Dd|jDn:t |j|D]$\}}|j|j%d|_dS)aG Restore the parameters stored with the `store` method. Useful to validate the model with EMA parameters without: affecting the original optimization process. Store the parameters before the `copy_to()` method. After validation (or model saving), use this to restore the former parameters. Args: parameters: Iterable of `torch.nn.Parameter`; the parameters to be updated with the stored parameters. If `None`, the parameters with which this `ExponentialMovingAverage` was initialized will be used. NzGThis ExponentialMovingAverage has no `store()`ed weights to `restore()`cg|] }|j Srr rs rrz$EMAModel.restore..Fr r!cg|] }|j Srr )rc_params rrz$EMAModel.restore..Fs6k6k6kw|6k6k6kr!)r RuntimeErrorrrrrryr)rrwrr|s rrestorezEMAModel.restore6s  " *kll l < /  444446k6kSWSj6k6k6k    #&d&=z"J"J / /   ....#'r!rjc.tj|}|d|j|_|jdks |jdkrt d|d|j|_t |jtst d|d|j|_t |jtst d|d |j |_ t |j tst d |d |j |_ t |j tst d |d |j |_ t |j ttfst d|d|j|_t |jttfst d|dd}|]||_t |jt st dt#d|jDst ddSdS)a Loads the ExponentialMovingAverage state. This method is used by accelerate during checkpointing to save the ema state dict. Args: state_dict (dict): EMA state. Should be an object returned from a call to :meth:`state_dict`. rrr$zDecay must be between 0 and 1rzInvalid min_decayrzInvalid optimization_steprzInvalid update_after_steprzInvalid use_ema_warmuprzInvalid inv_gammarz Invalid powerrNzshadow_params must be a listc3JK|]}t|tjVdSN)rurTensorrs r z+EMAModel.load_state_dict..|s.OOqz!U\22OOOOOOr!z!shadow_params must all be Tensors)copydeepcopyrrrHrrur*rintrrboolrrrrvall)rrjrs rrzEMAModel.load_state_dictOsM]:.. ^^GTZ88 :  tzC//<== =# T^DD$.%00 2011 1!+0CTE[!\!\$0#66 :899 9!+0CTE[!\!\$0#66 :899 9(nn-=t?RSS$-t44 7566 6# T^DD$.5#,77 2011 1^^GTZ88 $*ucl33 ._-- -"==  $!.D d0$77 A !?@@@OOD'4''''2.F$.F4.F.F.F.F.F.Fr!r)r$)NNNr$)7rr'rrrtypingrrrrrrr numpyrrmodelsr schedulersr utilsr rrrrrrrrrrpeftr torchvisionrrr)r r5r1rJr%r*rcrrfloat32rrr}rrrrrrr!rr<s DDDDDDDDDDDDDDDDDD ((((((&&&&&& *EEGG/......'&&&&&&)3))))("""J)3))))h(+3#3# 3##3#|3# < 3# < 3# L 3#!<3# %3# 8EL !8EL#9 9:3#3#3#3#l3S%,=N8O&V[Ub--ehotEHO7L&L M----"]#u|+,]69]IN]]]](tx   '* 8= QV kp    ,S$ $ $ $LFLFLFLFLFLFLFLFLFLFr!