3j SSKrSSKJr SSKrSSKrSSKJs Jr SSK J r SSK J r JrJrJr SSKJrJr SSKJr SSKJrJr SS KJr SS KJr SS KJrJrJ r SS K!J"r" SS K#J$r$ SSK%J&r& \"5(a SSK'J(s J)r* Sr+OSr+\RX"\-5r.Sr/Sr0SSjr1Sr2S Sjr3S!Sjr4S"S\5S-S\6\Rn-S-S\8\5S-S\8\9S-4Sjjr:"SS\5r;g)#N)Callable)Image) BertModel BertTokenizerQwen2TokenizerQwen2VLForConditionalGeneration)MultiPipelineCallbacksPipelineCallback)VaeImageProcessor)AutoencoderKLMagvitEasyAnimateTransformer3DModel)DiffusionPipeline)FlowMatchEulerDiscreteScheduler)is_torch_xla_availableloggingreplace_example_docstring) randn_tensor)VideoProcessor)EasyAnimatePipelineOutputTFaY Examples: ```python >>> import torch >>> from diffusers import EasyAnimateControlPipeline >>> from diffusers.pipelines.easyanimate.pipeline_easyanimate_control import get_video_to_video_latent >>> from diffusers.utils import export_to_video, load_video >>> pipe = EasyAnimateControlPipeline.from_pretrained( ... "alibaba-pai/EasyAnimateV5.1-12b-zh-Control-diffusers", torch_dtype=torch.bfloat16 ... ) >>> pipe.to("cuda") >>> control_video = load_video( ... "https://huggingface.co/alibaba-pai/EasyAnimateV5.1-12b-zh-Control/blob/main/asset/pose.mp4" ... ) >>> prompt = ( ... "In this sunlit outdoor garden, a beautiful woman is dressed in a knee-length, sleeveless white dress. " ... "The hem of her dress gently sways with her graceful dance, much like a butterfly fluttering in the breeze. " ... "Sunlight filters through the leaves, casting dappled shadows that highlight her soft features and clear eyes, " ... "making her appear exceptionally elegant. It seems as if every movement she makes speaks of youth and vitality. " ... "As she twirls on the grass, her dress flutters, as if the entire garden is rejoicing in her dance. " ... "The colorful flowers around her sway in the gentle breeze, with roses, chrysanthemums, and lilies each " ... "releasing their fragrances, creating a relaxed and joyful atmosphere." ... ) >>> sample_size = (672, 384) >>> num_frames = 49 >>> input_video, _, _ = get_video_to_video_latent(control_video, num_frames, sample_size) >>> video = pipe( ... prompt, ... num_frames=num_frames, ... negative_prompt="Twisted body, limb deformities, text subtitles, comics, stillness, ugliness, errors, garbled text.", ... height=sample_size[0], ... width=sample_size[1], ... control_video=input_video, ... ).frames[0] >>> export_to_video(video, "output.mp4", fps=8) ``` c[U[R5(aI[RRR UR S5USSS9RS5nO[U[R5(a0URUSUS45n[R"U5nOn[U[R5(aD[R"U5RUSUS45n[R"U5nO [S5e[U[R5(d8[R"U5R!SSS5R#5S- nU$) z\ Preprocess a single image (PIL.Image, numpy.ndarray, or torch.Tensor) to a resized tensor. rbilinearFsizemode align_cornersrzKUnsupported input type. Expected PIL.Image, numpy.ndarray, or torch.Tensor.go@) isinstancetorchTensornn functional interpolate unsqueezesqueezerresizenparrayndarray fromarray ValueError from_numpypermutefloat)image sample_sizes v/home/wildlama/miniconda3/lib/python3.13/site-packages/diffusers/pipelines/easyanimate/pipeline_easyanimate_control.pypreprocess_imager3]s#%&&##// OOA [zQV0 '!*  E5;; ' ' k!nk!n=> E2:: & &&--{1~{1~.NOfgg eU\\ * *  '//1a8>>@5H LcUGb>UVs/sH n[XRS9PM nn[R"U5SUnURSSSS5R S5nUb[X2S9n[R "US:SS 5nUR S5R S 5R/S Q5R S5n[R "USSUR5SSS/5nURURUR5nO2[R"USS2SS245nS USS2SS2SS24'OS upUb-[XBS9nURSSSS5R S5nOSnXU4$s snf) N)r1rrrr )rg?)r rrrNN) r3r stackr.r%wheretilertodevicedtype zeros_like) input_video num_framesr1validation_video_mask ref_imageframeinput_video_masks r2get_video_to_video_latentrGxsU`aU`E'GU` akk+.{ ; "))!Q15??B ,$45J$] !${{+@;+NPSUXY  099!<FFrJRRS_`jjklm $zz*:Q;CSCSCUVWCXZ[]^<_` /22;3E3E{GXGXY $// ArrE0BC (+ Q1W %(2% $YA %%aAq1;;A>  ) 33=bsE=cUnUnUupVXV- nXtU- :aUn[[XE- U-55n OUn [[X6- U-55n[[XH- S- 55n [[X9- S- 55n X4X-X-44$)Ng@)intround) src tgt_width tgt_heighttwthhwr resize_height resize_widthcrop_top crop_lefts r2get_resize_crop_region_for_gridrWs B B DA AG} 5!,-  E"&1*-. 5",345HE2,345I  8#;Y=U"V VVr4cUR[[SUR55SS9nUR[[SUR55SS9nXU- -nX%-SU- U--nU$)a Rescales `noise_cfg` tensor based on `guidance_rescale` to improve image quality and fix overexposure. Based on Section 3.4 from [Common Diffusion Noise Schedules and Sample Steps are Flawed](https://huggingface.co/papers/2305.08891). Args: noise_cfg (`torch.Tensor`): The predicted noise tensor for the guided diffusion process. noise_pred_text (`torch.Tensor`): The predicted noise tensor for the text-guided diffusion process. guidance_rescale (`float`, *optional*, defaults to 0.0): A rescale factor applied to the noise predictions. Returns: noise_cfg (`torch.Tensor`): The rescaled noise prediction tensor. rT)dimkeepdim)stdlistrangendim) noise_cfgnoise_pred_textguidance_rescalestd_textstd_cfgnoise_pred_rescaleds r2rescale_noise_cfgres{"""tE!_5I5I,J'KUY"ZHmmU1inn%= >mMG#''9: 6!>N:NR[9[[I r4c UR5nU(a[USS5nSUS'[R"USS2SS2SS2SS2SS24USSS9n[USS5nUSS- US'USS:waA[R"USS2SS2SS2SS2SS24USSS9n[R "XV/SS9nU$UnU$[USS5n[R"XSSS9nU$)Nrrr trilinearFrrY)rr\Fr$r cat)masklatentprocess_first_frame_only latent_size target_sizefirst_frame_resizedremaining_frames_resized resized_masks r2 resize_maskrss++-K;qr?+  Amm AqsAq ! +UZ ;qr?+ $Q!+ A q>Q '(}}Q12q!^$;[X]( $!99&9%TZ[\L  /L ;qr?+ }}T+]bc r4num_inference_stepsr> timestepssigmasc UbUb [S5eUbS[[R"UR5R R 55;nU(d[SURS35eUR"S X2S.UD6 URn[U5nX14$UbS[[R"UR5R R 55;nU(d[SURS35eUR"S XBS.UD6 URn[U5nX14$UR"U4S U0UD6 URnX14$) a Calls the scheduler's `set_timesteps` method and retrieves timesteps from the scheduler after the call. Handles custom timesteps. Any kwargs will be supplied to `scheduler.set_timesteps`. Args: scheduler (`SchedulerMixin`): The scheduler to get timesteps from. num_inference_steps (`int`): The number of diffusion steps used when generating samples with a pre-trained model. If used, `timesteps` must be `None`. device (`str` or `torch.device`, *optional*): The device to which the timesteps should be moved to. If `None`, the timesteps are not moved. timesteps (`list[int]`, *optional*): Custom timesteps used to override the timestep spacing strategy of the scheduler. If `timesteps` is passed, `num_inference_steps` and `sigmas` must be `None`. sigmas (`list[float]`, *optional*): Custom sigmas used to override the timestep spacing strategy of the scheduler. If `sigmas` is passed, `num_inference_steps` and `timesteps` must be `None`. Returns: `tuple[torch.Tensor, int]`: A tuple where the first element is the timestep schedule from the scheduler and the second element is the number of inference steps. zYOnly one of `timesteps` or `sigmas` can be passed. Please choose one to set custom valuesruzThe current scheduler class zx's `set_timesteps` does not support custom timestep schedules. Please check whether you are using the correct scheduler.)rur>rvzv's `set_timesteps` does not support custom sigmas schedules. Please check whether you are using the correct scheduler.)rvr>r>) r,setinspect signature set_timesteps parameterskeys __class__rulen) schedulerrtr>rurvkwargsaccepts_timesteps accept_sigmass r2retrieve_timestepsrs}>!3tuu'3w/@/@AXAX/Y/d/d/i/i/k+ll .y/B/B.CDab  M)MfM'' !)n  ))   C(9(9):Q:Q(R(](](b(b(d$ee .y/B/B.CD_`  GvGG'' !)n  ))  3MFMfM''  ))r4c48^\rSrSrSrSr/SQrS\S\\ -S\ \ -S\ S \ 4 U4S jjrSSjrSr\S5r\S5r\S 5r\S!5r \S"5r!\RD"5\#"\$5S S#S$S$S S S S%S&S S S'S S S S S S S(S S S)/S'S 4S\\\-S*\S -S+\S -S,\S -S-\RJS.\RJS/\RJS0\S -S1\&S -S\\\-S -S\S -S2\&S -S3\RN\\RN-S -S)\R(S -S\R(S -S\R(S -S\R(S -S\R(S -S4\S -S5\S6\(\\/S 4\)-\*-S -S7\\S8\&S9\\S -40S:jj55r+S;r,U=r-$)?EasyAnimateControlPipelinei a~ Pipeline for text-to-video generation using EasyAnimate. This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods the library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.) EasyAnimate uses one text encoder [qwen2 vl](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct) in V5.1. Args: vae ([`AutoencoderKLMagvit`]): Variational Auto-Encoder (VAE) Model to encode and decode video to and from latent representations. text_encoder (`~transformers.Qwen2VLForConditionalGeneration`, `~transformers.BertModel` | None): EasyAnimate uses [qwen2 vl](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct) in V5.1. tokenizer (`~transformers.Qwen2Tokenizer`, `~transformers.BertTokenizer` | None): A `Qwen2Tokenizer` or `BertTokenizer` to tokenize text. transformer ([`EasyAnimateTransformer3DModel`]): The EasyAnimate model designed by EasyAnimate Team. scheduler ([`FlowMatchEulerDiscreteScheduler`]): A scheduler to be used in combination with EasyAnimate to denoise the encoded image latents. ztext_encoder->transformer->vae)latents prompt_embedsnegative_prompt_embedsvae text_encoder tokenizer transformerrc>[TU]5 URUUUUUS9 [USS5b URR R OSUl[USS5bURROSUl [USS5bURROSUl [URS9Ul [URSSSS 9Ul[URS9Ulg) N)rrrrrrTr)vae_scale_factorF)r do_normalize do_binarizedo_convert_grayscale)super__init__register_modulesgetattrrconfigenable_text_attention_maskrspatial_compression_ratiovae_spatial_compression_ratiotemporal_compression_ratiovae_temporal_compression_ratior image_processormask_processorrvideo_processor)selfrrrrrrs r2r#EasyAnimateControlPipeline.__init__9s  %#  t]D1=    # # > > ' 3:$t2L2XDHH . .^_ *4;43M3YDHH / /_` + 1$BdBde/!??!%   .t?a?abr4rTNpromptnum_images_per_promptdo_classifier_free_guidancenegative_promptrrprompt_attention_masknegative_prompt_attention_maskr>r?max_sequence_lengthc U =(d URRn U =(d URRn Ub[U[5(aSn O3Ub![U[ 5(a [ U5n OURSn UGc[U[5(a SSUS./S./n OUVs/sH nSSUS./S.PM n nU Vs/sHoRRU/SSS 9PM nnURUS U SSS S S 9nURURR5nURnURnUR(aURUUSS9RSnO [S5eUR!US5nURXS9nURunnnUR!SUS5nUR#UU-US5nURU S9nU(Ga UGcUb [U[5(a SSUS./S./n OUVs/sH nSSUS./S.PM n nU Vs/sHoRRU/SSS 9PM nnURUS U SSS S S 9nURURR5nURnURnUR(aURUUSS9RSnO [S5eUR!US5nU(aUURSnURXS9nUR!SUS5nUR#X-US5nURU S9nXVXx4$s snfs snfs snfs snf)as Encodes the prompt into text encoder hidden states. Args: prompt (`str` or `list[str]`, *optional*): prompt to be encoded device: (`torch.device`): torch device dtype (`torch.dtype`): torch dtype num_images_per_prompt (`int`): number of images that should be generated per prompt do_classifier_free_guidance (`bool`): whether to use classifier free guidance or not negative_prompt (`str` or `list[str]`, *optional*): The prompt or prompts not to guide the image generation. If not defined, one has to pass `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. prompt_attention_mask (`torch.Tensor`, *optional*): Attention mask for the prompt. Required when `prompt_embeds` is passed directly. negative_prompt_attention_mask (`torch.Tensor`, *optional*): Attention mask for the negative prompt. Required when `negative_prompt_embeds` is passed directly. max_sequence_length (`int`, *optional*): maximum sequence length to use for the prompt. rrusertext)typer)rolecontentFT)tokenizeadd_generation_prompt max_lengthrightpt)rpaddingr truncationreturn_attention_mask padding_sidereturn_tensors) input_idsattention_maskoutput_hidden_stateszLLM needs attention_mask)r?r>r8r>)rr?r>rstrr\rshaperapply_chat_templater=rrr hidden_statesr,repeatview)rrrrrrrrrr>r?r batch_sizemessages_promptmr text_inputstext_input_idsbs_embedseq_len__negative_prompts r2 encode_prompt(EasyAnimateControlPipeline.encode_prompt`sZ0**0034,,33  *VS"9"9J  Jvt$<$<VJ&,,Q/J  &#&&!'-3V$D#E$*  $*!'-3W$E#F$* nvmuhi22A3^b2cmu ..$.&*$#)K&..):):)A)ABK(22N$/$>$> !.. $ 1 1,=Rim!2!-!$ !!;<<$9$@$@AVXY$Z !%((u(D ,22'1%,,Q0EqI %**86K+KWVXY 5 8 8 8 G '+A+I*z/3/O/O!'-3_$M#N-<  -<(!'-3=M$N#O-< nvmuhi22A3^b2cmu ..$.&*$#)K&..):):)A)ABK(22N-8-G-G *..)-):):,#A)-*;* - *$& !!;<<-K-R-RShjk-l * &,2215G%;%>%>U%>%Z "%;%B%B1F[]^%_ "%;%@%@Acelnp%q "-K-N-NV\-N-] *6KkkAXs5M%M"M'7%M,cnS[[R"URR5R R 55;n0nU(aX$S'S[[R"URR5R R 55;nU(aXS'U$)Neta generator)ryrzr{rstepr}r~)rrr accepts_etaextra_step_kwargsaccepts_generators r2prepare_extra_step_kwargs4EasyAnimateControlPipeline.prepare_extra_step_kwargss s7#4#4T^^5H5H#I#T#T#Y#Y#[\\  '*e $(3w/@/@ATAT/U/`/`/e/e/g+hh -6k *  r4c ^US-S:wd US-S:wa[SUSUS35eU bW[U4SjU 55(d=[STRSU V s/sHoTR;dMU PM sn 35eUbUb[S US US 35eUcUc [S 5eUbA[U[5(d,[U[ 5(d[S [ U535eUbUc [S5eUbUb[SUSUS 35eUbUc [S5eUbEUbAURUR:wa&[SURSURS35egggs sn f)Nrz8`height` and `width` have to be divisible by 16 but are z and .c3@># UHoTR;v M g7fN)_callback_tensor_inputs).0krs r2 :EasyAnimateControlPipeline.check_inputs..#sF 7Y!-- -7Ysz2`callback_on_step_end_tensor_inputs` has to be in z , but found zCannot forward both `prompt`: z and `prompt_embeds`: z2. Please make sure to only forward one of the two.zeProvide either `prompt` or `prompt_embeds`. Cannot leave both `prompt` and `prompt_embeds` undefined.z2`prompt` has to be of type `str` or `list` but is zEMust provide `prompt_attention_mask` when specifying `prompt_embeds`.z'Cannot forward both `negative_prompt`: z and `negative_prompt_embeds`: zWMust provide `negative_prompt_attention_mask` when specifying `negative_prompt_embeds`.zu`prompt_embeds` and `negative_prompt_embeds` must have the same shape when passed directly, but got: `prompt_embeds` z != `negative_prompt_embeds` )r,allrrrr\rr) rrheightwidthrrrrr"callback_on_step_end_tensor_inputsrs ` r2 check_inputs'EasyAnimateControlPipeline.check_inputss4 B;! urzQWX^W__dejdkklmn n - 9#F 7YF C C DTEaEaDbbn|^pH|^vwkokGkGbGpq|^pHoIJ   -";08N}o^00 ^ 5w  FC)@)@TZ\`IaIaQRVW]R^Q_`a a  $)>)Fde e  &+A+M9/9JK*++]_  " -2P2Xvw w  $)?)K""&<&B&BB --:-@-@,AB.445Q8C*L $7pHs E01E0c U bU RXvS9$UUUS- UR-S-X@R-XPR-4n [U[5(a*[ U5U:wa[ S[ U5SUS35e[XXvS9n [URS5(aXRR-n U $)Nr>r?rz/You have passed a list of generators of length z+, but requested an effective batch size of z@. Make sure the batch size matches the length of the generators.)rr>r?init_noise_sigma) r=rrrr\rr,rhasattrrr) rrnum_channels_latentsrBrrr?r>rrrs r2prepare_latents*EasyAnimateControlPipeline.prepare_latentsJs  ::V:9 9  !^ C C Ca G 88 8 77 7   i & &3y>Z+GA#i.AQR&<'gi  u&V 4>>#5 6 6 ? ??Gr4c UbURXvS9nSn /n [SURSU 5HHn XX-n URR U 5Sn U R 5n U R U 5 MJ [R"U SS9nXRRR-nUbURXvS9nSn /n[SURSU 5HHn X,X-nURR U5SnUR 5nUR U5 MJ [R"USS9nUURRR-nUU4$SnUU4$)Nrrrrh) r=r]rrencoderappendr rjrscaling_factor)rcontrol control_imagerrrr?r>rrbs new_controli control_bsnew_control_pixel_valuescontrol_pixel_values_bscontrol_image_latentss r2prepare_control_latents2EasyAnimateControlPipeline.prepare_control_latentsds{  jjj >>G  $),,F,HMB') $1m11!4b9*7AF*C'*.((//:Q*RST*U'*A*F*F*H'(//0GH : %*II.FA$N !$9DHHOOr?rrrrrrrtext_encoder_indexcpu)muT)rmrr r)rr)r?rhr)totalscale_model_input)encoder_hidden_statescontrol_latentsrr6)rarrrrrl)videor)frames)>rr r tensor_inputsrIrrr rrr\rr_execution_devicerr?rrr XLA_AVAILABLErrrrurrlatent_channelsrrsr rjr=r preprocessr.reshapefloat32rr@rrorderr progress_bar enumeraterrr#tensorchunkrerlocalspopupdatexm mark_stepdecode_latentsrpostprocess_videomaybe_free_model_hooksr)4rrrBrrrrrDrtrrrrrrrrrrrrrrrarurr>r?timestep_devicercontrol_video_latentsr%channels height_video width_videoref_image_latentsref_image_latents_conv_inrnum_warmup_stepsr0rtlatent_model_inputt_expand noise_predrnoise_pred_uncondr`callback_kwargsrcallback_outputsr&s4 r2__call__#EasyAnimateControlPipeline.__call__s< f *-=?U,V W W1E1S1S .flb()Ub[B&'       " ! * .  .!1  *VS"9"9J  Jvt$<$<VJ&,,Q/J''    (%%++E$$**E   "7(,(H(H+'#9"7+I     " ! * =#O$O dnn&E F F-? 3_iTU. *I*.@ 3_i. *I*NN,,  $xx>>&& . .          +$/0Dhl$m !$9A$= !:>:Z:Z 01A56`ub  &JWJ]J] GJ*lK 00;;%%aAq!4<<+X|[ <M*,,5==,AM)11*j(TZbjj1aAM%)$@$@00 % % !;?:Z:Z 01A56`ub %*$4$4W$=$@$@$O !:>:Z:Z 01A56`ub   JS// GJ*lK,,77!!!Q1a088j9PRZ\hjuv8I " 5== 9I!))*j(FZbbcdfgijlmopqI $ < <##00 ! ! ).(8(8(A %||~a A%6G)!Q(333 459:.b & $ii:S(TZ[\O(-(8(8(A %33 459:.b & $ii:S(TZ[\O!::9J  + +!II'=}&MNM$)II/MOd.e$f !&(((7 5 8 8 8 Gy>,?..BVBV,VV!)n   %8  9\!),1>>BFAaAaUYYy1}%=gn"4>>+>??)-)I)IJ\^_)`&!<<.@.F.Fq.I(IRXY\\,22]"--&*7$3 % .   ??$Q'488??+J+JJ$.$4$4QA$4$>MJ339C9I9I!9L6%!2^YjGj5k!kJ338H38N!2:aq!rJ..--j!WmHYmglmnop'3&(O?-3Xa[*@';D!Q'X$.229gFG$4$8$8-$XM-=-A-ABZ\r-s*I**A9I/IqSTuX\XfXfXlXlNlpqNq '') =LLNe-:lh&''0E((::T_:`EE ##%8O(66E: 9sIc&-c&& c4) r rrrrrrrrr) rTNNNNNNN)NNNNNNr).__name__ __module__ __qualname____firstlineno____doc__model_cpu_offload_seqrr rrrrrrrrr\rIboolr r!r>r?rrrrrpropertyrrarrrno_gradrEXAMPLE_DOC_STRING FloatTensorr/ Generatorrr r rK__static_attributes__ __classcell__)rs@r2rr s*=T$c $c6 A$c"M1 $c 3 $c 3 $cT&',026-16:59>B&*$(#&`ld3i`l #`l&* `l tCy4/ `l ||d* `l!& t 3`l %||d2`l). t(;`l t#`l{{T!`l!`lF!,#"'++/4nnr4!.F$$&& ((## ]]_12#'!# +/26'+*,'*26,-DH'+-16:59>B"' nr9B "%&*3F7d3iF7$JF7d F7 Tz F7 (( F7$//F7$$F7!4ZF7 F7tCy4/F7 #TzF7T\F7??T%//%::TAF7$F7 ||d*!F7"!& t 3#F7$ %||d2%F7&). t(;'F7(4Z)F7*+F7,'Sz4'78;KKNddgkk-F7.-1I/F70 1F729t#3F73F7r4rr9)r6)T)NNNN)r\r/rrrxr4r2rms  B0H99OO--6))MM   H %'T6!4JW&68'+(,"&!% 8*t8* %,,  %8*Cy4 8* K$  8*vF 7!2F 7r4