3j/BRSSKrSSKJr SSKrSSKrSSKrSSKJrJ r J r J r J r J r Jr SSKJr SSKJr "SS\R$5r"S S \R$5rS r\R,\ R.\ R0S4S jr\R,4S jr\R,4Sjrg)N)Optional)CrossAttention FeedForwardLTXFrequenciesPrecision LTXRopeTypegenerate_freq_grid_npinterleaved_freqs_cissplit_freqs_cis)nn)POS_EMBEDDING_EXP_VALUEScb^\rSrSrSrSU4SjjrSS\R4SjjrSr U=r $) BasicTransformerBlock1Da A basic Transformer block. Parameters: dim (`int`): The number of channels in the input and output. num_attention_heads (`int`): The number of heads to use for multi-head attention. attention_head_dim (`int`): The number of channels in each head. dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use. activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to be used in feed-forward. attention_bias (: obj: `bool`, *optional*, defaults to `False`): Configure if the attentions should contain a bias parameter. upcast_attention (`bool`, *optional*): Whether to upcast the attention computation to float32. This is useful for mixed precision training. norm_elementwise_affine (`bool`, *optional*, defaults to `True`): Whether to use learnable elementwise affine parameters for normalization. standardization_norm (`str`, *optional*, defaults to `"layer_norm"`): The type of pre-normalization to use. Can be `"layer_norm"` or `"rms_norm"`. norm_eps (`float`, *optional*, defaults to 1e-5): Epsilon value for normalization layers. qk_norm (`str`, *optional*, defaults to None): Set to 'layer_norm' or `rms_norm` to perform query and key normalization. final_dropout (`bool` *optional*, defaults to False): Whether to apply a final dropout after the last feed-forward layer. ff_inner_dim (`int`, *optional*): Dimension of the inner feed-forward layer. If not provided, defaults to `dim * 4`. ff_bias (`bool`, *optional*, defaults to `True`): Whether to use bias in the feed-forward layer. attention_out_bias (`bool`, *optional*, defaults to `True`): Whether to use bias in the attention output layer. use_rope (`bool`, *optional*, defaults to `False`): Whether to use Rotary Position Embeddings (RoPE). ffn_dim_mult (`int`, *optional*, defaults to 4): Multiplier for the inner dimension of the feed-forward layer. c r>[T U]5 [UUUSUUUU S9Ul[ UUSUUU S9Ulg)N) query_dimheadsdim_head context_dimapply_gated_attentiondtypedevice operationsT)dim_outglurrr)super__init__rattn1rff) selfdimn_headsd_headrattn_precisionrrrr __class__s R/home/wildlama/comfy/ComfyUI/custom_nodes/ComfyUI-LTXVideo/embeddings_connector.pyr BasicTransformerBlock1D.__init__3sV $"7!    !  returnc[RRRU5nUR S5nUR XBUS9nXQ-nUR S:XaUR S5n[RRRU5nURU5nXa-nUR S:XaUR S5nU$)Nr )maskpe)comfyldm common_ditrms_normsqueezerndimr)r hidden_statesattention_maskr,norm_hidden_states attn_output ff_outputs r&forwardBasicTransformerBlock1D.forwardXs #YY11::=I/77:jj!3RjP #3    ")11!4M#YY11::=IGG./ !1    ")11!4Mr()rr)NNFNNN)NN) __name__ __module__ __qualname____firstlineno____doc__rtorch FloatTensorr9__static_attributes__ __classcell__r%s@r&rrs:D## JeFWFWr(rc^\rSrSrSrSSSSSSS/S SS S S S S S 4S \\4U4S jjjrS rSr Sr SSjr SS\ RS\\ R4SjjrSrU=r$)Embeddings1DConnectoruTig@r FNnum_learnable_registersc >[TU]5 XlXlX@lXC-UlXlX`lXplXl Xl [R"[U5Vs/sHn[UR UUUU U U U S9PM sn5UlXC-nXlUR (a@[R""[$R&"UR UXS9S-S- 5Ulggs snf)N)rrrrrrrg@?)rrr out_channelsnum_attention_heads inner_dimcausal_temporal_positioningpositional_embedding_thetapositional_embedding_max_pos split_ropedouble_precision_roper ModuleListrangertransformer_1d_blocksrK Parameterr@randlearnable_registers)r in_channelscross_attention_dimattention_head_dimrP num_layersrSrTrRrKrrrrrUrVkwargs_rQr%s r&rEmbeddings1DConnector.__init__xs&  '#6 ,A+F(*D',H)$%:"%']]z* +A(NN'& 3*?!) + & " (< '>$  ' '')|| 00)5 (D $ (# s&$C2c[R"[S5Vs/sHnUSS2U4URU- PM snSS9nU$s snf)Nr )r!)r@stackrXrT)r indices_gridifractional_positionss r&get_fractional_positions.Embeddings1DConnector.get_fractional_positionss]${{q !AQT"T%F%Fq%II!   $#  s#A ctURnU[R[R4;a[ROUnUR U5nUR (a.[URURSUR5OURX$UR5RURS9nUS:Xa4XeRS5-RSS5R!S5nU$XeRS5S-S- -RSS5R!S5nU$)Nr rexp_2rerJ)rr@bfloat16float16float32rjrVrrSshaperQgenerate_freq_gridrto unsqueeze transposeflatten)r rgspacing source_dtyperriindicesfreqss r&precompute_freqs&Embeddings1DConnector.precompute_freqss&#))  >> MM  $<<\J)) "//""1%  ((9M9T9TU "(//" 0  g 99"==2r"  ::2>BQFG2r"   r(c URnURnSnSU-nSnUn US:Xa[R"[X#S9n OUS:Xa/SU[R "SXGUS9U- -- n U R US 9n ORUS :Xa[R"XXG-X2S 9n O3US :Xa-[R"US-U S-XG-X2S 9R5n W [R-S- n U $) Nr rJexprMrnrNrrm)rlinear)rrsqrt) rQrSr@tensorr arangerulinspacermathpi) r ryrrr!theta n_pos_dimsn_elemstartendr{s r&rt(Embeddings1DConnector.generate_freq_gridsnn// Z e ll#;5XG  Eell1c&&QTW&WXXGjjuj-G  nnCM&G nnq#q&#-df DGG#a'r(cfURnSnURX5nUR(a1US-nURSnXg- n[ XXUR 5upO[ XSU-5upU RUR5U RUR5UR4$)NrJre) rQr}rUrsr rPr rur) r rgryr!rr|expected_freqs current_freqspad_sizecos_freqsin_freqs r&precompute_freqs_cis*Embeddings1DConnector.precompute_freqs_cissnn%%l< ?? AXN!KKOM%5H!0!9!9" Hh"7uFl!K H{{4::& DJJ(?PPr(r4r5cUR(GaURSUR-S:Xd%SURSSURS35eURSUR-n[R"URUS45R UR 5nURS5RS5RS5S:R5nUSS2UR5R5SS24nURSnURSU- n[RRRUSSSU4SS 9n [R"US/S 9n X-SU - U--n[R"US UR UR S 9n[R""URS[R$UR S 9n U SSSS24n UR'U 5n [)UR*5H upU"XU S 9nM [,R.R0R3U5nX4$)ag The [`Transformer2DModel`] forward method. Args: hidden_states (`torch.LongTensor` of shape `(batch size, num latent pixels)` if discrete, `torch.FloatTensor` of shape `(batch size, channel, height, width)` if continuous): Input `hidden_states`. indices_grid (`torch.LongTensor` of shape `(batch size, 3, num latent pixels)`): attention_mask ( `torch.Tensor`, *optional*): An attention mask of shape `(batch, key_tokens)` is applied to `encoder_hidden_states`. If `1` the mask is kept, otherwise if `0` it is discarded. Mask will be converted into a bias, which adds large negative values to the attention scores corresponding to "discard" tokens. Returns: If `return_dict` is True, an [`~models.transformer_2d.Transformer2DModelOutput`] is returned, otherwise a `tuple` where the first element is the sample tensor. r rzHidden states sequence length z. must be divisible by num_learnable_registers .regN)padvalue)dimsgrM)r5r,)rKrsr@tiler\rurr2rvintboolr functionalrflip full_likerrrrr enumeraterYr.r/r0r1)r r4r5num_registers_duplicationsr\attention_mask_binarynon_zero_hidden_states non_zero_nums pad_lengthadjusted_hidden_states flipped_maskrg freqs_cis block_idxblocks r&r9Embeddings1DConnector.forwards~,  ' ' '##A&)E)EEJ V/ 0C0CA0F/GGuvzwSwSvTTUV VJ##A&$*F*FF '#(**((+Eq*I#b%%& &&q)11!4>>rBgMce "&3(0027791<& "388;M&,,Q/-?J%*XX%8%8%<%<&Q1j,A&=& "!::&;1#FL5|#'::;  #__$**%,, N||    "%-- @T@T $D$M2 --l; !*$*D*D E I!M!F ,,55mD ,,r() rRrVrrQr\rPrKrOrTrSrUrY)r)N)r;r<r=r> _supports_gradient_checkpointingrrrrjr}rtrr@Tensorr9rBrCrDs@r&rFrFus'+$ #*&'S$)14##!6"*#66p$B6 Q$26N-||N-!.N-N-r(rFzmodel.diffusion_model.c UR5VVs0sH,upxURU5(dMU[U5SU_M. n nn[U 5S:Xag[RR X3SS9n [ USUSUSURSS5UU UbUOS /U[R:HU[R:HS 9 n U RU 5 U $s snnf) NrT)disable_fast_fp8rPr_r`rFr ) rPr_r`rrrrTrUrV) items startswithlenr.opspick_operationsrFgetrSPLITrFLOAT64load_state_dict) sdconnector_prefixconnector_configr rope_typefrequencies_precision pe_max_poskv sd_connectorr connectors r&load_embeddings_connectorr]sHHJDA <<( ) &#  !"A%  <A**5$*OJ%,-BC+,@A#L1.223JER3=3IZPQs 1 1137N7V7VV Il+ -s CCc v[R"U5n[R"U5nURSS/5n[S3n[S3n[S3U;aUOUnURSS5URSS 5URS S 5URS S 5S.n [ UUU UUUU5$)N&connector_positional_embedding_max_posr zembeddings_connector.zvideo_embeddings_connector.z audio_adaln_single.linear.weightconnector_num_attention_headsrIconnector_attention_head_dimrHconnector_num_layersrJconnector_apply_gated_attentionFrPr_r`r)r from_dictrr _PREFIX_BASEr) rtransformer_configrrrrvideo_only_connector_prefixav_connector_prefixprefixrs r&load_video_embeddings_connectorrs%%&89I3==>PQ#''(PSTRUVJ%1N2G"H)N*EF^; < B  (  255 +R 144 *C ),,-CQG!3!7!7 -u"   %   r(c [R"U5n[R"U5nURSS/5nURSURSS55URSURSS55URS URS S 55URS S 5S.n[ U[ S3UUUUU5$)Nrr #audio_connector_num_attention_headsrrI"audio_connector_attention_head_dimrrHaudio_connector_num_layersrrJrFrzaudio_embeddings_connector.)rrrrrr)rrrrrrrs r&load_audio_embeddings_connectorrs%%&89I3==>PQ#''(PSTRUVJ 255 1  " "#BB G 144 0  " "#A3 G ),, (  " "#91 = "4!7!7 -u" $ % .34  r()rtypingrcomfy.ldm.common_ditr. comfy.opsr@comfy.ldm.lightricks.modelrrrrrr r r pos_embedding_exp_valuesr ModulerrFrrp INTERLEAVEDFLOAT32rrrr(r&rs  >]bii]@^-BII^-J(  ..%%199DCH.."JCH..r(