3jx SSKJr SSKrSSKJr SSKJrJr SSKJr SSK J r J r SS K J r SS KJr SS KJr SS KJr SS KJr \R,"\5rS\S\S\S\S\R4S\R64 SjrS\R6S\R6S\R64Sjr"SS5r"SS\R>\ 5r "SS\R>5r!"SS\R>5r""S S!\R>5r#"S"S#\R>5r$"S$S%\R>5r%S&\R6S\S\R64S'jr&S(\R6S\S)\R6S\R64S*jr'"S+S,\\\ 5r(g)-)AnyN)nn) ConfigMixinregister_to_config)logging)AttentionMixinAttentionModuleMixin)dispatch_attention_fn)get_timestep_embedding)Transformer2DModelOutput) ModelMixin)RMSNorm batch_sizeheightwidth patch_sizedevicereturnc.[R"X-X#-SUS9n[R"X-US9SS2S4US'[R"X#-US9SSS24US'URX-X#--S5R S5R USS5$)aX Generates 2D patch coordinate indices for a batch of images. Args: batch_size (`int`): Number of images in the batch. height (`int`): Height of the input images (in pixels). width (`int`): Width of the input images (in pixels). patch_size (`int`): Size of the square patches that the image is divided into. device (`torch.device`): The device on which to create the tensor. Returns: `torch.Tensor`: Tensor of shape `(batch_size, num_patches, 2)` containing the (row, col) coordinates of each patch in the image grid. r )rN.r.rr)torchzerosarangereshape unsqueezerepeat)rrrrrimg_idss g/home/wildlama/miniconda3/lib/python3.13/site-packages/diffusers/models/transformers/transformer_prx.py get_image_idsr#!s,kk&.0CQvVGll6#7G4PGFOll5#6vFtQwOGFO ??F0U5HI1 M W WXY Z a ablnoqr ssxq freqs_cisc.UR5R"/URSSQSPSPSP76nURURUR S9nUSUS-USUS--nUR"UR6R U5$)a Applies rotary positional embeddings (RoPE) to a query tensor. Args: xq (`torch.Tensor`): Input tensor of shape `(..., dim)` representing the queries. freqs_cis (`torch.Tensor`): Precomputed rotary frequency components of shape `(..., dim/2, 2)` containing cosine and sine pairs. Returns: `torch.Tensor`: Tensor of the same shape as `xq` with rotary embeddings applied. Nrr rdtyperr)floatrshapetorr*type_as)r%r&xq_xq_outs r" apply_roper1=s ((*   6bhhsm 6R 6 6A 6C BIISYY ?I v V ,y/@3v;/N NF >>288 $ , ,R 00r$c \rSrSrSrSrSrSrSSSS\RS\RS-S \RS-S \RS-S \R4 S jjr S r g)PRXAttnProcessor2_0Rz Processor for implementing PRX-style attention with multi-source tokens and RoPE. Supports multiple attention backends (Flash Attention, Sage Attention, etc.) via dispatch_attention_fn. Ncl[[RRS5(d [ S5eg)Nscaled_dot_product_attentionzHPRXAttnProcessor2_0 requires PyTorch 2.0, please upgrade PyTorch to 2.0.)hasattrrr functional ImportError)selfs r"__init__PRXAttnProcessor2_0.__init__[s,uxx**,JKKhi iLr$attn PRXAttention hidden_statesencoder_hidden_statesattention_maskimage_rotary_embrc dUc [S5eURU5nURupn URXSURUR 5nUR SSSSS5nUSUSUSpn URU 5n URU 5n URU5nURupn URXSURUR 5nUR SSSSS5nUSUSnnURU5nUb[X5n [X5n [R"UU 4SS9n[R"UU 4SS9nSnUGb U Runn nn URSnUR5S:wa[S UR35eURS U:wa[S URS S U35eU Rn[R "UU4[R"US 9nUR%U[R"S9n[R"UU/S S9nUSS2SSSS24R'S URUS 5nU R)SS5nUR)SS5nUR)SS5n[+UUUUUR,UR.S9nURunn n!n"URUU U!U"-5nUR0S"U5n[3UR05S:aUR0S"U5nU$)ae Apply PRX attention using PRXAttention module. Args: attn: PRXAttention module containing projection layers hidden_states: Image tokens [B, L_img, D] encoder_hidden_states: Text tokens [B, L_txt, D] attention_mask: Boolean mask for text tokens [B, L_txt] image_rotary_emb: Rotary positional embeddings [B, 1, L_img, head_dim//2, 2, 2] NzLPRXAttnProcessor2_0 requires 'encoder_hidden_states' containing text tokens.rr rrdimz"Unsupported attention_mask shape: r(zattention_mask last dim z must equal text length r*rr)) attn_maskbackendparallel_config) ValueError img_qkv_projr,rheadshead_dimpermutenorm_qnorm_k txt_kv_proj norm_added_kr1rcatrFronesboolr-expand transposer _attention_backend_parallel_configto_outlen)#r:r=r?r@rArBkwargsimg_qkvBL_img_img_qimg_kimg_vtxt_kvL_txttxt_ktxt_vkvattn_mask_tensorbsl_imgl_txtrones_img joint_maskquerykeyvalue attn_outputrseq_len num_headsrNs# r"__call__PRXAttnProcessor2_0.__call___s,( ! (kl l##M2mm !//!Atzz4==I//!Q1a0%aj'!*gaje E" E"!!"78ll !!TZZG1aA.ay&)u!!%(  'u7Eu7E IIuen! , IIuen! ,   %#kkOB5!KKNE!!#q( #EnFZFZE[!\]]##B'50 #;Nz PRX-style attention module that handles multi-source tokens and RoPE. Similar to FluxAttention but adapted for PRX's architecture. N query_dimrMdim_headbiasout_biasepsc>[TU]5 X lX0lX2-UlXl[ R"XS-US9Ul[URUSS9Ul [URUSS9Ul [ R"XS-US9Ul [URUSS9Ul [ R"/5UlURR![ R"URXS95 URR![ R""S55 UcUR%5nUR'U5 g)NrrT)relementwise_affiner )superr;rMrN inner_dimrrLinearrLrrPrQrRrS ModuleListr[appendDropout_default_processor_cls set_processor) r:rrMrrrr processor __class__s r"r;PRXAttention.__init__s   !)"IIiQTJdmmN dmmN 99YA DI#DMMstTmmB'  299T^^YNO 2::c?+  335I 9%r$r?r@rArBrc 2UR"UU4UUUS.UD6$)N)r@rArB)r)r:r?r@rArBr]s r"forwardPRXAttention.forwards5~~   #8)-     r$) rNrMrLrrSrQrPrr[rR)@FFư>Nrz)r{r|r}r~rr3r_available_processorsintrVr+r;rrrr __classcell__rs@r"r>r>s 101 &&& &  &  &&&H6:.204  ||  %||d2  t+   ,,-     r$r>c^\rSrSrSrS\S\S\\4U4SjjrS\RS\S\S\R4S jr S \RS\R4S jr S r U=r $) PRXEmbedNDa N-dimensional rotary positional embedding. This module creates rotary embeddings (RoPE) across multiple axes, where each axis can have its own embedding dimension. The embeddings are combined and returned as a single tensor Args: dim (int): Base embedding dimension (must be even). theta (int): Scaling factor that controls the frequency spectrum of the rotary embeddings. axes_dim (list[int]): list of embedding dimensions for each axis (each must be even). rFthetaaxes_dimcF>[TU]5 XlX lX0lgN)rr;rFrr)r:rFrrrs r"r;PRXEmbedND.__init__ s   r$posrcUS-S:XdeURRS:HnURRS:HnU(dU(a[RO[Rn[R "SUSXaRS9U- nSX7-- nUR S5UR S5-n [R"[R"U 5[R"U 5*[R"U 5[R"U 5/SS9n U R"/U RSSQSPSP76n U R5$) Nr rmpsnpurGg?r(rE) rtyperfloat32float64rrstackcossinrr,r+) r:rrFris_mpsis_npur*scaleomegaouts r"ropePRXEmbedND.ropesQw!||E)E)"(F  QQeJJG#Mu|$mmB%//!"44kk599S>EIIcN?EIIcNEIIVYN[acdkk0399Sb>010a0yy{r$idsc  URSn[R"[U5Vs/sH6o0R USS2SS2U4UR UUR 5PM8 snSS9nURS5$s snf)Nr(rEr)r,rrTrangerrrr)r:rn_axesiembs r"rPRXEmbedND.forward#sp2iiLQRXM ZMqYYs1a7|T]]1%5tzz BM Z }}Q [s=B)rrFr)r{r|r}r~rrlistr;rrrrrrrs@r"rrso !C!!tCy!  3su||  5<< ELL  r$rcr^\rSrSrSrS\S\4U4SjjrS\RS\R4Sjr S r U=r $) MLPEmbedderi,aF A simple 2-layer MLP used for embedding inputs. Args: in_dim (`int`): Dimensionality of the input features. hidden_dim (`int`): Dimensionality of the hidden and output embedding space. Returns: `torch.Tensor`: Tensor of shape `(..., hidden_dim)` containing the embedded representations. in_dim hidden_dimc>[TU]5 [R"XSS9Ul[R "5Ul[R"X"SS9Ulg)NTr)rr;rrin_layerSiLUsilu out_layer)r:rrrs r"r;MLPEmbedder.__init__;s?  &4@ GGI :Er$xrc`URURURU555$r)rrr)r:rs r"rMLPEmbedder.forwardAs#~~dii a(89::r$)rrr r{r|r}r~rrr;rrrrrrs@r"rr,s> FsFF ;;%,,;;r$rc ^\rSrSrSrS\4U4SjjrS\RS\ \ \R\R\R4\ \R\R\R444Sjr Sr U=r $) ModulationiEa Modulation network that generates scale, shift, and gating parameters. Given an input vector, the module projects it through a linear layer to produce six chunks, which are grouped into two tuples `(shift, scale, gate)`. Args: dim (`int`): Dimensionality of the input vector. The output will have `6 * dim` features internally. Returns: ((`torch.Tensor`, `torch.Tensor`, `torch.Tensor`), (`torch.Tensor`, `torch.Tensor`, `torch.Tensor`)): Two tuples `(shift, scale, gate)`. rFc.>[TU]5 [R"USU-SS9Ul[R R URRS5 [R R URRS5 g)NTrr) rr;rrlininit constant_weightr)r:rFrs r"r;Modulation.__init__Us^ 99S!c'5 $((//1- $((--+r$vecrcUR[RRU55SS2SSS24R SSS9n[ USS5[ USS54$)Nrr(rEr)rrr8rchunktuple)r:rrs r"rModulation.forward[s[hhr}}))#./4 ;AA!ALS!W~uSW~--r$)r) r{r|r}r~rrr;rrrrrrrs@r"rrEsr ,C, .<<. uU\\5<<=>ellTYT`T`bgbnbnFn@oo p..r$rc^\rSrSrSrSS\S\S\S\S-4U4SjjjrSS \RS \RS \RS \RS \RS-S\ \ \ 4S\R4Sjjr SrU=r$)PRXBlockibu Multimodal transformer block with text–image cross-attention, modulation, and MLP. Args: hidden_size (`int`): Dimension of the hidden representations. num_heads (`int`): Number of attention heads. mlp_ratio (`float`, *optional*, defaults to 4.0): Expansion ratio for the hidden dimension inside the MLP. qk_scale (`float`, *optional*): Scale factor for queries and keys. If not provided, defaults to ``head_dim**-0.5``. Attributes: img_pre_norm (`nn.LayerNorm`): Pre-normalization applied to image tokens before attention. attention (`PRXAttention`): Multi-head attention module with built-in QKV projections and normalizations for cross-attention between image and text tokens. post_attention_layernorm (`nn.LayerNorm`): Normalization applied after attention. gate_proj / up_proj / down_proj (`nn.Linear`): Feedforward layers forming the gated MLP. mlp_act (`nn.GELU`): Nonlinear activation used in the MLP. modulation (`Modulation`): Produces scale/shift/gating parameters for modulated layers. Methods: The forward method performs cross-attention and the MLP with modulation. N hidden_sizerv mlp_ratioqk_scalec >[TU]5 XlX lX-UlU=(d URS-Ul[ X-5UlXl[R"USSS9Ul [UUURSSS[5S9Ul[R"USSS9Ul[R "XRSS9Ul[R "XRSS9Ul[R "URUSS9Ul[R("SS9Ul[-U5Ulg) NgFrrr)rrMrrrrrrtanh) approximate)rr;rrvrNrrmlp_hidden_dimrr LayerNorm img_pre_normr>r3 attentionpost_attention_layernormr gate_projup_proj down_projGELUmlp_actr modulation)r:rrvrrrs r"r;PRXBlock.__init__s %"#0 4!4 !+"9:&LLTXY&!]])+ )+ [UZ`d(e%;0C0C%Pyy.A.AN 4#6#6 %Pww62 $[1r$r?r@tembrBrAr]rc fURU5upxUupn UupnSU -URU5-U -nURUUUUS9nXU--nSU -URU5-U -nXUR UR UR U55URU5-5--nU$)aG Runs modulation-gated cross-attention and MLP, with residual connections. Args: hidden_states (`torch.Tensor`): Image tokens of shape `(B, L_img, hidden_size)`. encoder_hidden_states (`torch.Tensor`): Text tokens of shape `(B, L_txt, hidden_size)`. temb (`torch.Tensor`): Conditioning vector used by `Modulation` to produce scale/shift/gates, shape `(B, hidden_size)` (or broadcastable). image_rotary_emb (`torch.Tensor`): Rotary positional embeddings applied inside attention. attention_mask (`torch.Tensor`, *optional*): Boolean mask for text tokens of shape `(B, L_txt)`, where `0` marks padding. **kwargs: Additional keyword arguments for API compatibility. Returns: `torch.Tensor`: Updated image tokens of shape `(B, L_img, hidden_size)`. r)r?r@rArB)rrrrrrrr)r:r?r@rrBrAr]mod_attnmod_mlp attn_shift attn_scale attn_gate mlp_shift mlp_scalemlp_gatehidden_states_modattn_outrs r"rPRXBlock.forwards@!OOD1,4) )0& h^t/@/@/OOR\\>>+"7)- " &H(<< ]d;;MJ JY V%DNN4<[TU]5 [R"USSS9Ul[R "XU-U-SS9Ul[R"[R"5[R "USU-SS95Ul g)NFrrTrr ) rr;rr norm_finalrlinear SequentialradaLN_modulation)r:rrrrs r"r;FinalLayer.__init__sn ,,{uRVWii *-D|-SZ^_ " bggi;PQT_P_fj9k lr$rrrcURU5RSSS9up4SUSS2SSS24-URU5-USS2SSS24-nURU5nU$)Nr rrE)r rrr )r:rrshiftrs r"rFinalLayer.forwardsi,,S177q7A  q$z" "dooa&8 85D!;L L KKNr$)r r rrrs@r"rrsQ4mCmSmm ELLU\\r$rimgcURup#pEUnURX#XF-XeU-U5n[R"SU5nURUSX6-U-5nU$)a Flattens an image tensor into a sequence of non-overlapping patches. Args: img (`torch.Tensor`): Input image tensor of shape `(B, C, H, W)`. patch_size (`int`): Size of each square patch. Must evenly divide both `H` and `W`. Returns: `torch.Tensor`: Flattened patch sequence of shape `(B, L, C * patch_size * patch_size)`, where `L = (H // patch_size) * (W // patch_size)` is the number of patches. znchpwq->nhwcpqr()r,rreinsum)rrbchwps r"img2seqrsdJA!A ++aAFAAvq 1C ,,' -C ++aQUQY 'C Jr$seqr,c[U[5(aUSSup4OS[U[R5(a[ US5[ US5pCO[ S[ U5S35eURupVnUnXxU--n URXSU-XH-XU5n[R"SU5nURXYX45nU$)a> Reconstructs an image tensor from a sequence of patches (inverse of `img2seq`). Args: seq (`torch.Tensor`): Patch sequence of shape `(B, L, C * patch_size * patch_size)`, where `L = (H // patch_size) * (W // patch_size)`. patch_size (`int`): Size of each square patch. shape (`tuple` or `torch.Tensor`): The original image spatial shape `(H, W)`. If a tensor is provided, the first two values are interpreted as height and width. Returns: `torch.Tensor`: Reconstructed image tensor of shape `(B, C, H, W)`. Nrrz shape type z not supportedznhwcpq->nchpwq) isinstancerrrrNotImplementedErrorrr,rr) rrr,rrrldrrs r"seq2imgr %s$%RSz1 E5<< ( (E!H s58}1!KU }N"KLLiiGA!A !e A ++aaq 1C ,,' -C ++aA !C Jr$c^\rSrSrSrSrSr\SS\S\S\S \S \ S \S \S \ S\S\ S\4U4Sjjj5r S\ RS\ RS\ R4SjrSS\ RS\ RS\ RS\ RS-S\\\4S-S\S\\ RS4\-4SjjrSrU=r$) PRXTransformer2DModeliNa- Transformer-based 2D model for text to image generation. Args: in_channels (`int`, *optional*, defaults to 16): Number of input channels in the latent image. patch_size (`int`, *optional*, defaults to 2): Size of the square patches used to flatten the input image. context_in_dim (`int`, *optional*, defaults to 2304): Dimensionality of the text conditioning input. hidden_size (`int`, *optional*, defaults to 1792): Dimension of the hidden representation. mlp_ratio (`float`, *optional*, defaults to 3.5): Expansion ratio for the hidden dimension inside MLP blocks. num_heads (`int`, *optional*, defaults to 28): Number of attention heads. depth (`int`, *optional*, defaults to 16): Number of transformer blocks. axes_dim (`list[int]`, *optional*): list of dimensions for each positional embedding axis. Defaults to `[32, 32]`. theta (`int`, *optional*, defaults to 10000): Frequency scaling factor for rotary embeddings. time_factor (`float`, *optional*, defaults to 1000.0): Scaling factor applied in timestep embeddings. time_max_period (`int`, *optional*, defaults to 10000): Maximum frequency period for timestep embeddings. Attributes: pe_embedder (`EmbedND`): Multi-axis rotary embedding generator for positional encodings. img_in (`nn.Linear`): Projection layer for image patch tokens. time_in (`MLPEmbedder`): Embedding layer for timestep embeddings. txt_in (`nn.Linear`): Projection layer for text conditioning. blocks (`nn.ModuleList`): Stack of transformer blocks (`PRXBlock`). final_layer (`LastLayer`): Projection layer mapping hidden tokens back to patch outputs. Methods: attn_processors: Returns a dictionary of all attention processors in the model. set_attn_processor(processor): Replaces attention processors across all attention layers. process_inputs(image_latent, txt): Converts inputs into patch tokens, encodes text, and produces positional encodings. compute_timestep_embedding(timestep, dtype): Creates a timestep embedding of dimension 256, scaled and projected. forward_transformers(image_latent, cross_attn_conditioning, timestep, time_embedding, attention_mask, **block_kwargs): Runs the sequence of transformer blocks over image and text tokens. forward(image_latent, timestep, cross_attn_conditioning, micro_conditioning, cross_attn_mask=None, attention_kwargs=None, return_dict=True): Full forward pass from latent input to reconstructed output image. Returns: `Transformer2DModelOutput` if `return_dict=True` (default), otherwise a tuple containing: - `sample` (`torch.Tensor`): Reconstructed image of shape `(B, C, H, W)`. z config.jsonTN in_channelsrcontext_in_dimrrrvdepthrr time_factortime_max_periodc T>[TU]5 UcSS/nXlX lURURS--UlXlXlXF-S:wa[SUSU35eXF-n [U5U :wa[SUSU 35eX@l X`l [XUS9Ul [R"URURS--URS S 9Ul[!S URS 9Ul[R"X0R5Ul[R&"[)U5V s/sH"n [+URURUS 9PM$ sn 5Ul[/URSUR5UlSUlgs sn f)N r rz Hidden size z must be divisible by num_heads zGot z but expected positional dim )rFrrTr)rr)rrF)rr;r#rrr&r'rKsumrrvr pe_embedderrrimg_inrtime_intxt_inrrrblocksr final_layergradient_checkpointing)r:r#rr$rrrvr%rrr&r'pe_dimrrs r"r;PRXTransformer2DModel.__init__s   BxH'$ ,,t/AA&.  "a '|K=8XYbXcde e) x=F "tH:-J6(ST T&"%&Qii 0 04??A3E EtGWGW^bc "#$:J:JK ii0@0@A mmu   &A $$NN' &     &d&6&64;L;LM&+# s)F%timestepr*rc UR[USURURSSS9R U55$)Nr*Tr) timesteps embedding_dim max_periodrflip_sin_to_cosdownscale_freq_shift)r.r r'r&r-)r:r5r*s r"_compute_timestep_embedding1PRXTransformer2DModel._compute_timestep_embeddingsD|| ""!//&& $%(  bi  r$r?r@rAattention_kwargs return_dict.c |URU5n[XR5nURU5nURupp[ XXRUR S9n URU 5nURX(RS9nURHZn[R"5(a3UR(a"URURUUUUU5nMPU"UUUUUS9nM\ UR!X5n[#XRUR5nU(dU4$[%US9$)aa Forward pass of the PRXTransformer2DModel. The latent image is split into patch tokens, combined with text conditioning, and processed through a stack of transformer blocks modulated by the timestep. The output is reconstructed into the latent image space. Args: hidden_states (`torch.Tensor`): Input latent image tensor of shape `(B, C, H, W)`. timestep (`torch.Tensor`): Timestep tensor of shape `(B,)` or `(1,)`, used for temporal conditioning. encoder_hidden_states (`torch.Tensor`): Text conditioning tensor of shape `(B, L_txt, context_in_dim)`. attention_mask (`torch.Tensor`, *optional*): Boolean mask of shape `(B, L_txt)`, where `0` marks padding in the text sequence. attention_kwargs (`dict`, *optional*): Additional arguments passed to attention layers. return_dict (`bool`, *optional*, defaults to `True`): Whether to return a `Transformer2DModelOutput` or a tuple. Returns: `Transformer2DModelOutput` if `return_dict=True`, otherwise a tuple: - `sample` (`torch.Tensor`): Output latent image of shape `(B, C, H, W)`. )rr)r*)r?r@rrBrA)sample)r/rrr-r,r#rr,r<r*r0ris_grad_enabledr2_gradient_checkpointing_funcrwr1r r)r:r?r5r@rAr>r?txtrrlrarrr!perblockoutputs r"rPRXTransformer2DModel.forwards*Fkk/0m__5kk#$)) qq__]MaMab   g &..xyy.I[[E$$&&4+F+F77NN" "%*-%'#1 !(s(oo}/B/BC9 'v66r$)r0r1r2rr-r#rvrrr,r&r.r'r/) r i ig @rIN'g@@rK)NNT)r{r|r}r~r config_name _supports_gradient_checkpointingrrr+rr;rrr*r<rrrrVrrrrrrs@r"r"r"Nsy<| K'+$"#$7,7,7, 7,  7,  7,7,7,7,7,7,7,7,r  ELL    Y^YeYe  "/326 K7||K7,,K7 %|| K7  t+ K7 sCx.4/ K7K7 u||S !$< <K7K7r$r"))typingrrrconfiguration_utilsrrutilsrrr r attention_dispatchr embeddingsr modeling_outputsrmodeling_utilsr normalizationr get_loggerr{loggerrrrr#r1r3Moduler>rrrrrrr r"ryr$r"rYs| B<6/7'#   H %tct3tsttUZUaUatfkfrfrt815<<1ELL1U\\1*kk\9 29929 z, , ^;"));2..:zryyzz%%P35<<<&&3&u||& &RS7J ^S7r$