3j}V SSKrSSKrSSKJr SSKrSSKJr SSKJrJ r SSK J r J r SSK JrJrJrJr SSKJr SS KJrJr SS KJrJrJr SS KJr SS KJr SS KJ r SSK!J"r" SSK#J$r$ SSK%J&r&J'r' \RP"\)5r*"SS5r+"SS5r,"SS\R RZ\5r."SS\RZ5r/\"SS\RZ55r0\"SS\$\\\ \ \55r1Sr2g)N)Any) ConfigMixinregister_to_config)FromOriginalModelMixinPeftAdapterMixin)apply_lora_scale deprecateis_torch_versionlogging)maybe_allow_in_graph)ContextParallelInputContextParallelOutput)AttentionMixinAttentionModuleMixin FeedForward)dispatch_attention_fn) CacheMixin)PixArtAlphaTextProjection)Transformer2DModelOutput) ModelMixin)AdaLayerNormSingleRMSNormc\rSrSrSrSrg)LTXVideoAttentionProcessor2_0(c6Sn[SSU5 [U0UD6$)Nz~`LTXVideoAttentionProcessor2_0` is deprecated and this will be removed in a future version. Please use `LTXVideoAttnProcessor`rz1.0.0)r LTXVideoAttnProcessor)clsargskwargsdeprecation_messages g/home/wildlama/miniconda3/lib/python3.13/site-packages/diffusers/models/transformers/transformer_ltx.py__new__%LTXVideoAttentionProcessor2_0.__new__)s)_17[T U]5 U S:wa [S5eX@lXB-UlUc UROXC-UlXlUbUOUUlX`lXPl Xl X l Sn Sn [RRXB-XS9Ul[RRXC-XS9Ul[RR#XRUS9Ul[RR#URUR US9Ul[RR#URUR US9Ul[RR+/5UlUR,R/[RR#URURUS95 UR,R/[RR1U55 U cUR35n UR5U 5 g)Nrms_norm_across_headszIOnly 'rms_norm_across_heads' is supported as a valid value for `qk_norm`.gh㈵>Tepselementwise_affine)ri)superr4NotImplementedErrorhead_dim inner_dim inner_kv_dimrerjuse_biasrhout_dimrIrannrrMrNLinearrJrKrL ModuleListrVappendDropout_default_processor_cls set_processor)r3rerIrfrgrhrirjrkrl processornorm_epsnorm_elementwise_affine __class__s r$r4LTXAttention.__init__ws  - -%&qr r !).6.>DNNHDW":M:Y#6_h     "&hh&&x'7X&r hh&&x':&u HHOOI~~DOI HHOOD$<$O>OVZO[ HHOOD$<$O>OVZO[ hh))"-  588??4>>4<J>41Q5I$!$>J~~d3HZjuntuu S Ks D.D D"D)rjrhrtrIrurvrNrMrxrerKrVrJrLrw) r@r?TNTrnNr_)r)r*r+r,rr~_available_processorsintfloatboolstrr4rarbrr- __classcell__rs@r$r7r7ss223 *..(&(&(& (&  (&  (&(&!4Z(&(&(&(&Z6:.204 v||v %||d2v t+ v  ,,- v vvr'r7c^\rSrSrSS\S\S\S\S\S\S\S S 4U4S jjjrS \S \S\S\S\\R\\4S\RS \R4Sjr SS\RS \S -S\S -S\S -S\\R\\4S -S\RS -S \\R\R44Sjjr Sr U=r$)LTXVideoRotaryPosEmbeddimbase_num_frames base_height base_width patch_size patch_size_tthetar<Ncv>[TU]5 XlX lX0lX@lXPlX`lXplg)N) rrr4rrrrrrr) r3rrrrrrrrs r$r4LTXVideoRotaryPosEmbed.__init__s6 .&$$( r'rW num_framesheightwidthrope_interpolation_scaledevicec[R"U[RUS9n[R"U[RUS9n[R"U[RUS9n [R"XUSS9n [R"U SS9n U R S5R USSSS5n UbU SS2SS24US-UR-UR- U SS2SS24'U SS2SS24US-UR-UR- U SS2SS24'U SS2SS24US-UR-UR- U SS2SS24'U RSS 5RSS5n U $) N)rUrij)indexingrrrErr)raarangefloat32meshgridstack unsqueezerepeatrrrrrrS transpose) r3rWrrrrrgrid_hgrid_wgrid_fgrids r$_prepare_video_coords,LTXVideoRotaryPosEmbed._prepare_video_coordssjfEMM&Ie5==Hj fM~~fftD{{4Q'~~a '' Aq!Q? # /1Q3<*B1*EEHYHYY\`\p\ppDAaCL1Q3<*B1*EEWZ^ZjZjjDAaCL1Q3<*B1*EEWZ^ZiZiiDAaCL||Aq!++Aq1 r'r8 video_coordsc URS5nUcURUUUUUURS9nOT[R"USS2S4UR - USS2S4UR - USS2S4UR- /SS9nSn URn UR[R"[R"XR5[R"XR5URS-UR[RS 9-n U [R-S - n XRS5S-S- -n U R!SS 5R#S5n U R%5R'SSS9n U R)5R'SSS9n URS-S:wa[R*"U SS2SS2SURS-245n[R,"U SS2SS2SURS-245n[R."X/SS9n [R."X/SS9n X4$) Nr)rrrErr>rg?)rrUg@)sizerrrarrrrrlinspacemathlogrrpirrrScosrepeat_interleavesin ones_like zeros_likecat)r3r8rrrrrrWrstartendfreqs cos_freqs sin_freqs cos_padding sin_paddings r$rLTXVideoRotaryPosEmbed.forwards#''*  --)A$++ .D;; A&)=)== A&)9)99 A&8  Djj enn HHUJJ ' HHS** % HHM ''--   #%+a/!34B'//2IIK11!1< IIK11!1< 88a<1 //)Aq.DHHqL.4H*IJK**9Q>TXX\>5I+JKK ;":CI ;":CI##r')rrrrrrr)rrErE@)NNNNN)r)r*r+r,rrr4tuplerarbrrrr-rrs@r$rrso "       (    #( eU(B"C   :"&! MQ,04$||4$$J4$d 4$ Tz 4$ #( eU(B"Cd"J 4$llT)4$ u||U\\) *4$4$r'rc4^\rSrSrSrSS\S\S\S\S\S\S \S \S \S \4U4S jjjr SS\ RS\ RS\ RS\ \ R\ R4S-S\ RS-S\ R4 Sjjr SrU=r$)LTXVideoTransformerBlockiai Transformer block used in [LTX](https://huggingface.co/Lightricks/LTX-Video). Args: dim (`int`): The number of channels in the input and output. num_attention_heads (`int`): The number of heads to use for multi-head attention. attention_head_dim (`int`): The number of channels in each head. qk_norm (`str`, defaults to `"rms_norm"`): The normalization layer to use. activation_fn (`str`, defaults to `"gelu-approximate"`): Activation function to use in feed-forward. eps (`float`, defaults to `1e-6`): Epsilon value for normalization layers. rnum_attention_headsattention_head_dimrjrl activation_fnattention_biasattention_out_biasrprqc :>[T U]5 [XU S9Ul[ UUUUUSUUS9Ul[XU S9Ul[ UUUUUUUUS9Ul[XS9Ul [R"[R"SU5US-- 5Ulg)Nro)rerIrfrgrirjrkrl)rerjrIrfrgrirkrl)rr?)rrr4rnorm1r7attn1norm2attn2rffry Parameterrarandnscale_shift_table) r3rrrrjrlrrrrprqrs r$r4!LTXVideoTransformerBlock.__init__-s S>PQ !%(' $'   S>PQ ! 3%(''   c?!#ekk!S.ACH.L!Mr'Nr8r9tembr;encoder_attention_maskr<cURS5nURU5nURRSnURSR UR 5UR XcRS5US5-n U RSS9upppUSU --U -nURUSUS9nUUU --nURUUSUS9nUU-nURU5SU--U -nURU5nUUU--nU$) NrNNrEr>rr)r8r9r;)r9r;r:) rrrrFrTrreshapeunbindrrrr)r3r8r9rr;rrWnorm_hidden_statesnum_ada_params ada_values shift_msa scale_msagate_msa shift_mlp scale_mlpgate_mlpattn_hidden_states ff_outputs r$r LTXVideoTransformerBlock.forwardXsH#''* !ZZ 6//55a8++J7::4;;G$,,  ! nbK  JTIZIZ_`IZIaF h9/1y=AIM!ZZ,"&-(  &(:X(EE !ZZ "7!1 (  &(:: !ZZ 6!i-H9TGG./ % H(<< r')rrrrrr)rngelu-approximateTTư>Fr)r)r*r+r,r`rrrrr4rarbrrr-rrs@r$rrs0//##'#()N )N!)N )N ! )N  )N)N)N!)N)N!)N)N`FJ6: %||% %||%ll %   ell :;dB % !& t 3 % %%r'rc":^\rSrSrSrSrS/rS/r\"SSSS 9\"SSSS 9\"SS SS 9S .\"SSSS 9\"SSSS 9S .\ "SSS 9S.r \ S.S\ S\ S\ S\ S\ S\ S\ S\ S\ S\ S\S\S\ S\S\SS4 U4S jjj5r\"S!5S/S"\R&S#\R&S$\R(S%\R&S&\ S-S'\ S-S(\ S-S)\\\\4\R&-S-S*\R&S-S!\\ \4S-S+\S\R&4S,jj5rS-rU=r$)0LTXVideoTransformer3DModelia A Transformer model for video-like data used in [LTX](https://huggingface.co/Lightricks/LTX-Video). Args: in_channels (`int`, defaults to `128`): The number of channels in the input. out_channels (`int`, defaults to `128`): The number of channels in the output. patch_size (`int`, defaults to `1`): The size of the spatial patches to use in the patch embedding layer. patch_size_t (`int`, defaults to `1`): The size of the tmeporal patches to use in the patch embedding layer. num_attention_heads (`int`, defaults to `32`): The number of heads to use for multi-head attention. attention_head_dim (`int`, defaults to `64`): The number of channels in each head. cross_attention_dim (`int`, defaults to `2048 `): The number of channels for cross attention heads. num_layers (`int`, defaults to `28`): The number of layers of Transformer blocks to use. activation_fn (`str`, defaults to `"gelu-approximate"`): Activation function to use in feed-forward. qk_norm (`str`, defaults to `"rms_norm_across_heads"`): The normalization layer to use. TnormrrErF) split_dim expected_dims split_outputr)r8r9r)rrE) gather_dimr)ropeproj_out in_channels out_channelsrrrrrj num_layersrrlrrcaption_channelsrrr<NcX>[TU]5 U=(d UnXV-n[R"UU5Ul[R "[ R"SU5US-- 5Ul[USS9Ul [U US9Ul [USSSUUSS 9Ul[R"[!U5Vs/sHn[#UUUUU U UUU U S 9 PM sn5Ul[R&"US SS 9Ul[R"UU5UlSUlgs snf) NrrF)use_additional_conditions) in_features hidden_sizerrr)rrrrrrr) rrrrjrlrrrrprqrro)rrr4ryrzproj_inrrarrr time_embedrcaption_projectionrr r{rangertransformer_blocks LayerNormnorm_outr gradient_checkpointing)r3r r rrrrrjrrrlrrrrrrurYrs r$r4#LTXVideoTransformer3DModel.__init__s,& #2{ '< yyi8 !#ekk!Y.G)UX..X!Y,YRWX";HXfo"p*!%  #%--z* +A)!(;'9(;#"/#1'9 '> + # $ YDUS  )\: &+#) sD'attention_kwargsr8r9timesteprrrrrr return_dictc URXXgX5n UbBURS:Xa2SURUR5- S-nUR S5nUR S5n UR U5nURUR5U URS9upURU SUR S55nURU SUR S55nURU5nURU SUR S55nURHPn[R"5(a)UR(aURUUUUU U5nMFU"UUUU US9nMR UR SUSS2SS2S4-nUSS2SS2S4USS2SS2S4nnUR#U5nUSU--U-nUR%U5nU (dU4$['US 9$) NrrEgr)rW hidden_dtyper>)r8r9rr;rr)sample)r ndimrTrUrrrrrSrHrrrais_grad_enabledr_gradient_checkpointing_funcrrr r)r3r8r9rrrrrrrrrr;rWrembedded_timestepblockscale_shift_valuesshiftscaleoutputs r$r"LTXVideoTransformer3DModel.forwards 99]Ogv " -2H2M2MQR2R&'*@*C*CMDWDW*X&X\d%d "%;%E%Ea%H ""''*  ]3 "&//    !&,,#2#  yyR27-22:rCTCYCYZ\C]^ $ 7 78M N 5 : ::r=K]K]^`Ka b,,E$$&&4+F+F $ A A!)$* ! !&"/*?%5+A ! -&"33J?BSTUWXZ^T^B__)!Q'24Fq!Qw4Ou m4 %U3e; }-9 'v66r') rrrrr r rrr)r-rErE rrrrnFriTT)NNNNNNT)r)r*r+r,r` _supports_gradient_checkpointing _skip_layerwise_casting_patterns_repeated_blocksrr_cp_planrrrrrr4r rarb LongTensorrdictrrr-rrs@r$rrsl4(,$(.x$232AQ]bc%9AUVej%k&:QVWfk&l $aqtT#aqtT *QaH H#%"$#'/.(- $##'!=,=,=, =,  =, ! =, =,!=,=,=,=,"&=,=,=,=, !!=," #=,=,~()"&! UY,026 @7||@7 %||@7"" @7 !& @7 $J @7d @7Tz@7#(ue(;"rr>r)rPrrarrSrrTrU)xrrrx_realx_imag x_rotatedouts r$rOrO2s|HC[[G,33B7NF fWf-26>>qAI 779s?Y__.4 4 8 8 AC Jr')3rrtypingrratorch.nnryconfiguration_utilsrrloadersrrutilsr r r r utils.torch_utilsr _modeling_parallelrr attentionrrrattention_dispatchr cache_utilsr embeddingsrmodeling_outputsrmodeling_utilsr normalizationrr get_loggerr)rrrModuler7rrrrOr(r'r$rLs   B?KK5LII6$27'7   H %66@@F=v588??$8=v@c$RYYc$LcryyccLn7 ^-CEUWan7n7br'