3j8SSKJr SSKrSSKJr SSKJrJr SSKJ r SSK J r SSK J r Jr SS KJrJr SS KJr SS KJr SS KJrJrJr \ R4"\5r"S S\R:5r"SS\\5rg))AnyN) ConfigMixinregister_to_config)logging)LuminaFeedForward) AttentionLuminaAttnProcessor2_0)&LuminaCombinedTimestepCaptionEmbeddingLuminaPatchEmbed)Transformer2DModelOutput) ModelMixin)LuminaLayerNormContinuousLuminaRMSNormZeroRMSNormc*^\rSrSrSrSS\S\S\S\S\S\S \S \S \S S 4U4SjjjrSS\ RS\ RS\ RS\ RS\ RS\ RS\ \ \ 4S -S \ R4SjjrSrU=r$)LuminaNextDiTBlock$a A LuminaNextDiTBlock for LuminaNextDiT2DModel. Parameters: dim (`int`): Embedding dimension of the input features. num_attention_heads (`int`): Number of attention heads. num_kv_heads (`int`): Number of attention heads in key and value features (if using GQA), or set to None for the same as query. multiple_of (`int`): The number of multiple of ffn layer. ffn_dim_multiplier (`float`): The multiplier factor of ffn layer dimension. norm_eps (`float`): The eps for norm layer. qk_norm (`bool`): normalization for query and key. cross_attention_dim (`int`): Cross attention embedding dimension of the input text prompt hidden_states. norm_elementwise_affine (`bool`, *optional*, defaults to True), dimnum_attention_heads num_kv_heads multiple_offfn_dim_multipliernorm_epsqk_normcross_attention_dimnorm_elementwise_affinereturnNc \>[T U]5 X-Ul[R"[ R "U/55Ul[USX-U(aSOSUUSSS[5S9 Ul [R"5URl [UUX-U(aSOSUUSSS[5S9 Ul [U[SU-S- 5UUS9Ul[#UUU S9Ul['XU S 9Ul['XU S 9Ul['XU S 9Ul['XU S 9Ulg) Nlayer_norm_across_headsh㈵>F) query_dimrdim_headrheadskv_headsepsbiasout_bias processorr)r inner_dimrr) embedding_dimrr)r'elementwise_affine)super__init__head_dimnn Parametertorchzerosgater r attn1Identityto_outattn2r int feed_forwardrnorm1r ffn_norm1norm2 ffn_norm2 norm1_context) selfrrrrrrrrr __class__s h/home/wildlama/miniconda3/lib/python3.13/site-packages/diffusers/models/transformers/lumina_nextdit2d.pyr0LuminaNextDiTBlock.__init__5s8 2 LL.A-B!CD  $/18-d%!,.   KKM  3/18-d%!,.   .%#+/*#1  '$;  !G^_SCZ[  G^_$%8[rs hidden_statesattention_maskimage_rotary_embencoder_hidden_states encoder_masktembcross_attention_kwargsc UnURX5uppUR"SU U UUUS.UD6n URU5nUR"SU UUUSS.UD6nXRR 5R SSSS5-nX-nURS5nURRS"U5nXRS5R 5URU5--nURURU5SU RS5--5nXRS5R 5URU5--nU$)a Perform a forward pass through the LuminaNextDiTBlock. Parameters: hidden_states (`torch.Tensor`): The input of hidden_states for LuminaNextDiTBlock. attention_mask (`torch.Tensor): The input of hidden_states corresponse attention mask. image_rotary_emb (`torch.Tensor`): Precomputed cosine and sine frequencies. encoder_hidden_states: (`torch.Tensor`): The hidden_states of text prompt are processed by Gemma encoder. encoder_mask (`torch.Tensor`): The hidden_states of text prompt attention mask. temb (`torch.Tensor`): Timestep embedding with text prompt embedding. cross_attention_kwargs (`dict[str, Any]`): kwargs for cross attention. )rGrJrHquery_rotary_embkey_rotary_embNr)r=r7rAr:r6tanhviewflattenr9 unsqueezer?r<r>r@)rBrGrHrIrJrKrLrMresidualnorm_hidden_statesgate_msa scale_mlpgate_mlpself_attn_outputnorm_encoder_hidden_statescross_attn_outputmixed_attn_output mlp_outputs rDforwardLuminaNextDiTBlock.forwardvst,!=AJJ}<[9i:: ,"4)-+  %  &*%7%78M%N" JJ ,"<'-  %  . 0@0E0EaBPQ0RR,@-55b9 ))!,->?  #5#5a#8#=#=#?$**]B[#[[ &&t~~m'DIL_L_`aLbHb'cd %(:(:1(=(B(B(Dt~~V`Ga(aa rF) r7r:r<r>r@r6r1r=rAr?)T)N)__name__ __module__ __qualname____firstlineno____doc__r;floatboolr0r4Tensordictstrrrc__static_attributes__ __classcell__rCs@rDrr$s 4)-?t ?t!?t ?t  ?t " ?t?t?t!?t"&?t ?t?tR9=9||9 9 ,, 9 %|| 9 ll 9ll9!%S#X 59 99rFrc ^\rSrSrSr/SQr\SS\S\S-S\S-S\S-S \S-S \S-S \S-S \S-S \S-S\S-S\ S-S\ S-S\S-S\S-SS4U4Sjjj5r SS\ RS\ RS\ RS\ RS\ RS\ \\4S\\ R\-4SjjrSrU=r$)LuminaNextDiT2DModela LuminaNextDiT: Diffusion model with a Transformer backbone. Inherit ModelMixin and ConfigMixin to be compatible with the sampler StableDiffusionPipeline of diffusers. Parameters: sample_size (`int`): The width of the latent images. This is fixed during training since it is used to learn a number of position embeddings. patch_size (`int`, *optional*, (`int`, *optional*, defaults to 2): The size of each patch in the image. This parameter defines the resolution of patches fed into the model. in_channels (`int`, *optional*, defaults to 4): The number of input channels for the model. Typically, this matches the number of channels in the input images. hidden_size (`int`, *optional*, defaults to 4096): The dimensionality of the hidden layers in the model. This parameter determines the width of the model's hidden representations. num_layers (`int`, *optional*, default to 32): The number of layers in the model. This defines the depth of the neural network. num_attention_heads (`int`, *optional*, defaults to 32): The number of attention heads in each attention layer. This parameter specifies how many separate attention mechanisms are used. num_kv_heads (`int`, *optional*, defaults to 8): The number of key-value heads in the attention mechanism, if different from the number of attention heads. If None, it defaults to num_attention_heads. multiple_of (`int`, *optional*, defaults to 256): A factor that the hidden size should be a multiple of. This can help optimize certain hardware configurations. ffn_dim_multiplier (`float`, *optional*): A multiplier for the dimensionality of the feed-forward network. If None, it uses a default value based on the model configuration. norm_eps (`float`, *optional*, defaults to 1e-5): A small value added to the denominator for numerical stability in normalization layers. learn_sigma (`bool`, *optional*, defaults to True): Whether the model should learn the sigma parameter, which might be related to uncertainty or variance in predictions. qk_norm (`bool`, *optional*, defaults to True): Indicates if the queries and keys in the attention mechanism should be normalized. cross_attention_dim (`int`, *optional*, defaults to 2048): The dimensionality of the text embeddings. This parameter defines the size of the text representations used in the model. scaling_factor (`float`, *optional*, defaults to 1.0): A scaling factor applied to certain parameters or layers in the model. This can be used for adjusting the overall scale of the model's operations. )patch_embeddernormffn_normN sample_size patch_size in_channels hidden_size num_layersrrrrr learn_sigmarrscaling_factorrcb>[TU]5 XlX lX0lU (aUS-OUUlX@lX`lXF-UlXl [X#USS9Ul [R"[R"U55Ul[#[%US5U S9Ul[R("[+U5Vs/sHn[-UUUUU U U U 5PM sn5Ul[1U[%US5SSSX"-UR -S9UlXF-S -S :XdS 5egs snf) NrT)ryrz embed_dimr(i)r{rFgư>)r-conditioning_embedding_dimr.r'r(out_dimrz+2d rope needs head dim to be divisible by 4)r/r0rxryrz out_channelsr{rr1r~r rur2r3r4empty pad_tokenr mintime_caption_embed ModuleListrangerlayersrnorm_out)rBrxryrzr{r|rrrrrr}rrr~_rCs rDr0LuminaNextDiT2DModel.__init__sD$ &$&/:K!O &#6 #: ,.!kX\ ekk+&>?"HK.DW# mmz* +A#' &' +   2%'*;'=$+d.?.??   2a71<k>kk<1 sD,rGtimesteprJrKrIrMc JURX5uppURUR5nURX#U5n UR 5nUR Hn U "UUUUUU US9nM UR X5nUR=pU SupURS5nX-X--nUSS2SU24RUX-X-XUR5nURSSSSSS5RSS5RSS5nU(dU4$[US 9$) a] Forward pass of LuminaNextDiT. Parameters: hidden_states (torch.Tensor): Input tensor of shape (N, C, H, W). timestep (torch.Tensor): Tensor of diffusion timesteps of shape (N,). encoder_hidden_states (torch.Tensor): Tensor of caption features of shape (N, D). encoder_mask (torch.Tensor): Tensor of caption masks of shape (N, L). )rLrMrNrQrrr)sample)rutodevicerrkrrrysizerVrpermuterWr)rBrGrrJrKrIrM return_dictmaskimg_sizerLlayer height_tokens width_tokensheightwidth batch_sizesequence_lengthoutputs rDrcLuminaNextDiT2DModel.forward#sK&;?:M:Mm:n7 X+..}/C/CD&&x U#((* [[E! %'=M! m: (,6   "''* !2u7LM%a)9/)9&9:?? /1F eievev &&q!Q1a8@@AFNNqRST9 'v66rF) r1r{rzrrrrrruryrxr~r)rri rNNr"TTig?)NT)rerfrgrhri _skip_layerwise_casting_patternsrr;rjrkr0r4rlrmrnrtuplerrcrorprqs@rDrsrss+Z(N$!""#"&!#*,#'"%+/!%#'#*.'*>l>l$J>l4Z >l 4Z >l $J >l!4Z>lDj>l4Z>l"DL>l$,>lD[>l>l!4Z>l >l !>l>lN2637||37,,37 %|| 37 ll 37  ,, 37!%S#X37 u|| 7 73737rFrs) typingrr4torch.nnr2configuration_utilsrrutilsr attentionr attention_processorr r embeddingsr r modeling_outputsrmodeling_utilsr normalizationrrr get_loggerreloggerModulerrsrTrFrDrsd B)C8'QQ   H %KK\d7:{d7rF