3j LSSKrSSKJr SSKJr SSKrSSKJr SSKJs J r SSK J r J r SSKJrJrJr SSKJrJrJr SS KJrJr SS KJrJr SS KJr SS KJr SS K J!r!J"r"J#r#J$r$ SSK%J&r& SSK'J(r( \RR"\*5r+\"SS\55r,"SS5r-"SS5r.SIS\R^S\R^S\R^S\0S\0S\-S-S\R^4Sjjr1S\2\R^S4S \2\R^S4S!\0S"\0S\2\R^S44 S#jr3S$\R^S%\R^S!\0S"\0S\R^4 S&jr4S'\R^S%\R^S(\0S!\0S"\0S\R^4 S)jr5SJSKS+jjr6SJSKS,jjr7SJSKS-jjr8"S.S/\Rr5r:"S0S1\Rr5r;"S2S35r<"S4S55r="S6S*\RRr\5r>"S7S85r?"S9S:5r@"S;S<\RRr\5rA"S=S>\Rr5rB"S?S@\Rr5rC"SASB\Rr5rD"SCSD\Rr5rE"SESF\Rr5rF"SGSH\&\ \\\\\5 rGg)LN) dataclass)Any) ConfigMixinregister_to_config)FluxTransformer2DLoadersMixinFromOriginalModelMixinPeftAdapterMixin) BaseOutputapply_lora_scalelogging)ContextParallelInputContextParallelOutput)AttentionMixinAttentionModuleMixin)dispatch_attention_fn) CacheMixin)TimestepEmbedding Timestepsapply_rotary_embget_1d_rotary_pos_embed) ModelMixin)AdaLayerNormContinuousc2\rSrSr%SrS\S'SrS\S'Srg) Flux2Transformer2DModelOutput+al The output of [`Flux2Transformer2DModel`]. Args: sample (`torch.Tensor` of shape `(batch_size, num_channels, height, width)`): The hidden states output conditioned on the `encoder_hidden_states` input. kv_cache (`Flux2KVCache`, *optional*): The populated KV cache for reference image tokens. Only returned when `kv_cache_mode="extract"`. z torch.TensorsampleNFlux2KVCache | Nonekv_cache)__name__ __module__ __qualname____firstlineno____doc____annotations__r __static_attributes__r!i/home/wildlama/miniconda3/lib/python3.13/site-packages/diffusers/models/transformers/transformer_flux2.pyrr+s &*H#*r)rc\rSrSrSrSrS\RS\R4SjrS\ \R\R44Sjr S r S r g ) Flux2KVLayerCache;zPer-layer KV cache for reference image tokens in the Flux2 Klein KV model. Stores the K and V projections (post-RoPE) for reference tokens extracted during the first denoising step. Tensor format: (batch_size, num_ref_tokens, num_heads, head_dim). c SUlSUlgNk_refv_refselfs r*__init__Flux2KVLayerCache.__init__Bs*. *. r)r1r2cXlX lg)zStore reference token K/V.Nr0)r4r1r2s r*storeFlux2KVLayerCache.storeFs   r)returncbURc [S5eURUR4$)z$Retrieve cached reference token K/V.z$KV cache has not been populated yet.)r1 RuntimeErrorr2r3s r*getFlux2KVLayerCache.getKs+ :: EF Fzz4::%%r)c SUlSUlgr/r0r3s r*clearFlux2KVLayerCache.clearQs  r)r0N) r"r#r$r%r&r5torchTensorr8tupler=r@r(r!r)r*r,r,;sJ /5<<  &U5<<56& r)r,cT\rSrSrSrS\S\4SjrS\S\4SjrS\S\4S jr S r S r g ) Flux2KVCacheVzContainer for all layers' reference-token KV caches. Holds separate cache lists for double-stream and single-stream transformer blocks. num_double_layersnum_single_layersc[U5Vs/sH n[5PM snUl[U5Vs/sH n[5PM snUlSUlgs snfs snfNr)ranger,double_block_cachessingle_block_cachesnum_ref_tokens)r4rHrI_s r*r5Flux2KVCache.__init__\sYAFGXAY#ZAYA$5$7AY#Z AFGXAY#ZAYA$5$7AY#Z #$$[#Zs AA  layer_idxr:c URU$r/)rMr4rRs r* get_doubleFlux2KVCache.get_doublea'' 22r)c URU$r/)rNrTs r* get_singleFlux2KVCache.get_singledrWr)cURHnUR5 M URHnUR5 M SUlgrK)rMr@rNrO)r4caches r*r@Flux2KVCache.cleargs<--E KKM.--E KKM.r))rMrOrNN) r"r#r$r%r&intr5r,rUrYr@r(r!r)r*rFrFVsG %#%#% 3C3,=33C3,=3 r)rFquerykeyvaluenum_txt_tokensrOr r:cUS:Xa Uc [XX&S9$UbnUR5upx[R"USS2SU24XqSS2US24/SS9n [R"USS2SU24XSS2US24/SS9n [X XS9$Un X4-n USS2SU 24n USS2X24nUSS2U S24nUSS2SU 24nUSS2X24nUSS2U S24nUSS2SU 24nUSS2X24nUSS2U S24n[R"X/SS9n[R"UUU/SS9n [R"UUU/SS9n [UXUS9nUSS2SU 24nUSS2U S24n[XXS9n[R"UUU/SS9$)aCausal attention for KV caching where reference tokens only self-attend. All tensors use the diffusers convention: (batch_size, seq_len, num_heads, head_dim). Without cache (extract mode): sequence layout is [txt, ref, img]. txt+img tokens attend to all tokens, ref tokens only attend to themselves. With cache (cached mode): sequence layout is [txt, img]. Cached ref K/V are injected between txt and img. rNbackenddim)rr=rBcat)r_r`rarbrOr rer1r2k_allv_all ref_startref_endq_txtq_refq_imgk_txtk_imgv_txtv_img q_txt_img attn_txt_imgattn_txtattn_imgattn_refs r*_flux2_kv_causal_attentionrzos$x/$UHH||~  3q/>/12Eq./?Q;RSYZ[ 5O^O!34e1noCU=VW]^_$U5JJI-G !ZiZ- E !Y&& 'E !WX+ E :I: E 9$$ %E 78 E !ZiZ- E !Y&& 'E !WX+ E 5.a0I IIueU+ 3E IIueU+ 3E(E'RLAz zM*HAyzM*H%U5JH 99h(3 ;;r) img_params. ref_paramsnum_refseq_lenc j/n[X5HupVURS:Xa"URS5nURS5nURSnUR [ R "URXrS5URXsS5SS2US2SS24/SS95 M [U5$)zSBlend modulation parameters so that the first `num_ref` positions use `ref_params`.rrfrNrg) zipndim unsqueezeshapeappendrBriexpandrD)r{r|r}r~blendedimrmBs r*_blend_mod_paramsrsGj- 77a<aBaB HHQK II1r*BIIa",EaSTn,UV  . >r)img_modref_modcvURS:Xa"URS5nURS5n[R"USSS9n[R"USSS9nUSSUSS4nUSSUSS4n/n[ Xg5H"up[ XX#5n UR U 5 M$ [R"USS9$)zBlend double-block image-stream modulations for a [ref, img] sequence layout. Takes raw modulation tensors (before `Flux2Modulation.split`) and returns a blended raw tensor that is compatible with `Flux2Modulation.split(mod, 2)`. rrfrrgrr)rrrBchunkrrextendri) rrr}r~ img_chunks ref_chunksimg_modsref_mods all_paramsimg_setref_setrs r*_blend_double_block_modsrs||q##A&##A&WaR0JWaR0J1QAa1H1QAa1HJ3#GgG'"4 99ZR ((r) single_modnum_txtc ^URS:Xa"URS5nURS5n[R"USSS9n[R"USSS9n/n[ XV5HupURS:Xa"URS5nU RS5n UR Sn UR XS5n U R XS5n UR[R"U SS2SU2SS24XSS2X#-S2SS24/SS95 M [R"USS9$)zBlend single-block modulations for a [txt, ref, img] sequence layout. Takes raw modulation tensors and returns a blended raw tensor compatible with `Flux2Modulation.split(mod, 1)`. rrfrrrgrN) rrrBrrrrrri) rrrr}r~r{r|rrrr im_expanded rm_expandeds r*_blend_single_block_modsrs!))!, ##A&Z3JWaR0JGj- 77a<aBaB HHQKiiB/ iiB/  IIQ!^,kq'J[J]_`G`;ab  . 99W" %%r)Flux2AttentioncURU5nURU5nURU5nS=n=pxUb@URb3UR U5nUR U5nUR U5nX4XVXx4$r/)to_qto_kto_vadded_kv_proj_dim add_q_proj add_k_proj add_v_proj attn hidden_statesencoder_hidden_statesr_r`ra encoder_query encoder_key encoder_values r*_get_projectionsrs IIm $E ))M "C IIm $E266M6K(T-C-C-O(=> oo&;< (=> u[ GGr)cURU5RSSS9up4nS=n=pxUb3[US5(a"URU5RSSS9upgnX4XVXx4$)Nrrrgr/ to_added_qkv)to_qkvrhasattrrrs r*_get_fused_projectionsr sy M2888CE299M9K(WT>-J-J484E4EF[4\4b4bcdjl4b4m1 M u[ GGr)cTUR(a [XU5$[XU5$r/)fused_projectionsrr)rrrs r*_get_qkv_projectionsrs' %d;PQQ D1F GGr)cf^\rSrSrSrU4SjrS\RS\R4SjrSr U=r $) Flux2SwiGLUiz Flux 2 uses a SwiGLU-style activation in the transformer feedforward sub-blocks, but with the linear projection layer fused into the first linear layer of the FF sub-block. Thus, this module has no trainable parameters. cV>[TU]5 [R"5Ulgr/)superr5nnSiLUgate_fn)r4 __class__s r*r5Flux2SwiGLU.__init__!s wwy r)xr:cRURSSS9up#URU5U-nU$)Nrrrg)rr)r4rx1x2s r*forwardFlux2SwiGLU.forward%s.# LL r !r))r) r"r#r$r%r&r5rBrCrr( __classcell__rs@r*rrs- !%,,r)rc ^\rSrSrS S\S\S-S\S\S-S\4 U4SjjjrS \RS \R4S jr S r U=r $)Flux2FeedForwardi+Nrhdim_outmult inner_dimbiasc>[TU]5 Uc [X-5nU=(d Un[R"XS-US9Ul[ 5Ul[R"XBUS9Ulg)Nrr) rr5r^rLinear linear_inract_fn linear_out)r4rhrrrrrs r*r5Flux2FeedForward.__init__,s]   CJI.S3A DA!m ))ITBr)rr:clURU5nURU5nURU5nU$r/)rrr)r4rs r*rFlux2FeedForward.forward>s0 NN1  KKN OOA r))rrr)N@NF) r"r#r$r%r^floatboolr5rBrCrr(rrs@r*rr+sz# $ C CtC C : C  CC$%,,r)rc \rSrSrSrSrSrS SSS\RS\RS\RS-S \RS-S \R4 S jjr S r g)Flux2AttnProcessoriENcr[[S5(d"[URRS35egNscaled_dot_product_attentionz; requires PyTorch 2.0. Please upgrade your pytorch version.rF ImportErrorrr"r3s r*r5Flux2AttnProcessor.__init__I3q899!8!8 99tuv v:r)rrrrattention_maskimage_rotary_embr:c [XU5upgppURSURS45nURSURS45nURSURS45nURU5nUR U5nUR bU RSURS45n U RSURS45n U RSURS45n UR U 5n URU 5n [R"X/SS9n[R"X/SS9n[R"X/SS9nUb[XeSS9n[XuSS9n[UUUUURURS9nURSS5nURUR 5nUbOUR#UR$SUR$SUR$S- /SS9up2UR'U5nUR(S"U5nUR(S"U5nUbX#4$U$) Nrrfrg sequence_dim attn_maskreparallel_configrrr)r unflattenheadsnorm_qnorm_kr norm_added_q norm_added_krBrirr_attention_backend_parallel_configflattentodtypesplit_with_sizesr to_add_outto_out) r4rrrrrr_r`rarrrs r*__call__Flux2AttnProcessor.__call__MsIH\ !6H DE+TZZ$45mmBR 01TZZ$45 E"kk#  ! ! -)33BR8HIM%//TZZ4DEK)33BR8HIM --m>TUCU&I$5G NN3q)G"334::Y>_>_>a b I %.1*<6E>KbKbMh &8+?6E1xQhQhM2(// $ 5 5 M&--a3 %((5 ,3@3Q3Q&,,Q/1D1DQ1GJ_JeJefgJh1hiop4R4 0 !%)OO4I$J ! A}5  A}5 , 7 7 r)r!)NNNNNrr"r#r$r%r&rrr5rBrCr,strr^rr(r!r)r*rrsw/3.204-1$(P!P!||P! %|| P!  t+ P!  ,,- P!$d*P!TzP!P! P!P!r)rc(^\rSrSr\r\\/rSS\S\S\S\ S\ S\S-S \ S-S \ S \ S \S \ 4U4Sjjjr SS\ RS\ RS-S\ RS-S\ RS-S\ R4 SjjrSrU=r$)riN query_dimrdim_headdropoutrradded_proj_biasout_biasepsout_dimelementwise_affinec >[T U]5 X0lU bU OX2-UlXlU bU OUUlU bX-OUUlXPlX@lX`l Xpl [RRXRUS9Ul[RRXRUS9Ul[RRXRUS9Ul[RR#X9U S9Ul[RR#X9U S9Ul[RR)/5UlUR*R-[RRURUR US95 UR*R-[RR/U55 Ub[RR#X9S9Ul[RR#X9S9Ul[RRX`RUS9Ul[RRX`RUS9Ul[RRX`RUS9Ul[RRURXS9UlU cUR=5n UR?U 5 g)Nrrr)r) rr5head_dimrrrruse_biasrrrrBrrrrrRMSNormrr ModuleListrrDropoutrrrrrr_default_processor_cls set_processor)r4rrrrrrrrrrr processorrs r*r5Flux2Attention.__init__s$  $+$7X=M"")"5w9 ,3,?W(U   !2.HHOOI~~DOI HHOOI~~DOI HHOOI~~DOI hh&&xM_&` hh&&xM_&` hh))"-  588??4>>4<J>41Q5I$!$>J~~d3HZjuntuu S Ks D.D D"D)rrrrrrrrrrrrrrrrrrrrr) @FNTTh㈵>NTNr)r"r#r$r%rrr_available_processorsr^rrr5rBrCrr(rrs@r*rrs&//1EF (,'+#'3&3&3& 3&  3&  3&:3&3&3&3&3&!3&3&p6:.204 v||v %||d2v t+ v  ,,- v vvr)c \rSrSrSrSrSrS SSS\RS\RS-S\RS-S \R4 S jjr S r g) Flux2ParallelSelfAttnProcessori8Ncr[[S5(d"[URRS35egrrr3s r*r5'Flux2ParallelSelfAttnProcessor.__init__<rr)rFlux2ParallelSelfAttentionrrrr:c URU5n[R"USUR-URUR -/SS9upVUR SSS9upxn URSURS45nURSURS45nU RSURS45n URU5nURU5nUb[XtSS9n[XSS9n[UUU UURURS9nURSS5nUR!UR"5nUR%U5n[R&"X&/SS9nUR)U5nU$)Nrrrgrfrrr)to_qkv_mlp_projrBsplitrmlp_hidden_dimmlp_mult_factorrrrrrrrrrrrr mlp_act_fnrir) r4rrrrqkvmlp_hidden_statesr_r`ras r*r'Flux2ParallelSelfAttnProcessor.__call__@su,,]; !& A.0C0CdFZFZ0Zc"   IIaRI0ETZZ$45mmBR 01TZZ$45 E"kk#  '$U1ME"3qIC-   $++ 11  &--a3 %((5 !OO,=> ="D"M  M2 r)r!NNrr!r)r*r9r98spw/304 -*-||- t+ -  ,,- -  --r)r9c\rSrSrSrSrSrSrSSSS\RS\RS-S \RS-S \ S-S \ S-S \ S \ S\R4Sjjr Srg) Flux2KVParallelSelfAttnProcessoripap Attention processor for Flux2 single-stream blocks with KV caching support for reference image tokens. When `kv_cache_mode` is "extract", reference token K/V are stored and causal attention is used. When `kv_cache_mode` is "cached", cached ref K/V are injected during attention. When no KV args are provided, behaves identically to `Flux2ParallelSelfAttnProcessor`. Ncr[[S5(d"[URRS35egrrr3s r*r5)Flux2KVParallelSelfAttnProcessor.__init__|rr)rr<rrrr rrbrOr:c .URU5n [R"U SUR-URUR -/SS9upU R SSS9upnU RSURS45n U RSURS45n URSURS45nURU 5n URU 5n Ub[XSS9n [XSS9n US:XaOUbLUS:aFUnXx-nURU SS2UU24R5USS2UU24R55 US:XaUS:a[XXXRS9nOBUS :XaUb[XXSXPRS 9nO"[!U U UUURUR"S 9nUR%S S5nUR'U R(5nUR+U 5n [R,"UU /SS9nUR/U5nU$) Nrrrgrfrrrrdrrrr)r>rBr?rr@rArrrrrrr8r rzrrrrrrrBrir)r4rrrrr rrbrOhidden_states_projrCrDr_r`rarlrm attn_outputs r*r)Flux2KVParallelSelfAttnProcessor.__call__s,"11-@!& T^^!3T5H5H4K_K_5_ `fh"  IIaRI0ETZZ$45mmBR 01TZZ$45 E"kk#  '$U1ME"3qIC I %(*>>TUCU&I$5G NN3q)G"334::Y>_>_>a b I %.1*<4E>KbKbKh &8+?4E1xQhQhK0(// $ 5 5 K"))!Q/ !nnU[[1 !OO,=> ;0A"BK  M2 r)r!)NNNNrrr r!r)r*rHrHpsw/304-1$(A*A||A t+ A  ,,- A $d* ATzAAA AAr)rHc^\rSrSrSr\r\\/rSr SS\ S\ S\ S\ S \ S \ S \ S \ S \ S\ S\ 4U4Sjjjr SS\RS\RS-S\RS-S\R4SjjrSrU=r$)r<ia Flux 2 parallel self-attention for the Flux 2 single-stream transformer blocks. This implements a parallel transformer block, where the attention QKV projections are fused to the feedforward (FF) input projections, and the attention output projections are fused to the FF output projections. See the [ViT-22B paper](https://arxiv.org/abs/2302.05442) for a visual depiction of this type of transformer block. FNrrrrrrrrr mlp_ratiorAc >[T U]5 X0lUbUOX2-UlXlUbUOUUlUbX-OUUlXPlX@lXl [XR-5Ul Xl [RRURURS-URUR--US9Ul[#5Ul[RR'X7U S9Ul[RR'X7U S9Ul[RRURUR-UR US9UlU cUR/5n UR1U 5 g)Nrrr)rr5rrrrrrrrPr^r@rArBrrr>rrBrrrrrr)r4rrrrrrrrrrPrArrs r*r5#Flux2ParallelSelfAttention.__init__sQ  $+$7X=M"")"5w9 ,3,?W(U   "!)nn"<=. %xx NNDNNQ.1D1DtG[G[1[[bf / &-hh&&xM_&` hh&&xM_&` hhoodnnt7J7J&JDLL_goh  335I 9%r)rrrr:c [[R"URR5R R 55nUR5VVs/sHupgXe;dM UPM nnn[U5S:a:[RSUSURRRS35 UR5VV s0sHupiXe;dM Xi_M nnn UR"XX#40UD6$s snnfs sn nfr"r#) r4rrrr-r.r/rPr0r1s r*r"Flux2ParallelSelfAttention.forwardsg//0G0GHSSXXZ['-||~R~tq9Q~ R }  ! NN)-8MdnnNfNfNoNoMpqFG $*<<>J>41Q5I$!$>J~~d>^W]^^ S Ks C?.C? D"D)rrrrrBr@rArPrrrrrr>r) r3r4r5FTr6NTg@rNrF)r"r#r$r%r&r9rrHr7_supports_qkv_fusionr^rrr5rBrCrr(rrs@r*r<r<s<;=]^  #' -&-&-& -&  -&  -&-&-&-&!-&-&-&-&d/304 _||_ t+_ ,,- _  __r)r<cF^\rSrSrSS\S\S\S\S\S\4 U4SjjjrSS \RS \RS -S \RS \ \R\R4S -S\ \ \ 4S -S\S\S -S\ \R\R44SjjrSrU=r$)Flux2SingleTransformerBlockirhnum_attention_headsattention_head_dimrPrrc >[TU]5 [R"USUS9Ul[ UUUUUUUUS[ 5S9 Ulg)NFrrr) rrrrrrrrPrAr)rr5r LayerNormnormr<r9rr4rhrXrYrPrrrs r*r5$Flux2SingleTransformerBlock.__init__sS LLCH /'%46   r)Nrrtemb_modrjoint_attention_kwargssplit_hidden_states text_seq_lenr:cUb%URSn[R"X!/SS9n[R US5Supn UR U5n SU -U -U-n U=(d 0nUR "SU US.UD6n XU --nUR[R:XaURSS5nU(aUSS2SU24USS2US24pX!4$U$)Nrfrgr)rr r!) rrBriFlux2Modulationr?r]rrfloat16clip) r4rrr`rrarbrc mod_shift mod_scalemod_gatenorm_hidden_statesrMs r*r#Flux2SingleTransformerBlock.forward0s ! ,066q9L!II'<&LRSTM)8)>)>x)KA)N& h!YY}5)m/AAIM!7!=2ii ,- % &;(>>   %-- /)..vu=M 3@M\MAQ3RTabceqerbrTs=(7 7 r))rr]rư>F)NNFNr"r#r$r%r^rrr5rBrCrDdictr rrr(rrs@r*rWrWs   !         DFJ8<$)#'$!||$! %||d2$!,, $!   ell :;dB $! !%S#X 5 $!"$!Dj$! u||U\\) *$!$!r)rWcF^\rSrSrSS\S\S\S\S\S\4 U4SjjjrSS \RS \RS \RS \RS\ \R\R4S -S\ \ \ 4S -S\ \R\R44SjjrSrU=r$)Flux2TransformerBlockiWrhrXrYrPrrc >[TU]5 [X-5Ul[R "USUS9Ul[R "USUS9Ul[UUUUUUUUU[5S9 Ul [R "USUS9Ul [XXFS9Ul [R "USUS9Ul[XXFS9Ulg)NFr[) rrrrrrrrrr)rhrrr)rr5r^r@rr\norm1 norm1_contextrrrnorm2rff norm2_context ff_contextr^s r*r5Flux2TransformerBlock.__init__Xs !#/2\\#%SI \\#%SQ"!'% (*   \\#%SI "siS\\#%SQ*si[r)Nrr temb_mod_img temb_mod_txtrrar:cU=(d 0n[RUS5uupxn upn [RUS5uupnunnnURU5nSU-U-U-nURU5nSU-U-U -nUR"SUUUS.UD6nUunnU U-nUU-nUR U5nUSU --U -nUR U5nXU--nUU-nUU-nURU5nUSU--U-nURU5nUUU--nUR[R:XaURSS5nX!4$)Nrrf)rrrrerfr!) rgr?rvrwrrxryrzr{rrBrhri)r4rrr}r~rra shift_msa scale_msagate_msa shift_mlp scale_mlpgate_mlp c_shift_msa c_scale_msa c_gate_msa c_shift_mlp c_scale_mlp c_gate_mlprmnorm_encoder_hidden_statesattention_outputsrMcontext_attn_output ff_outputcontext_ff_outputs r*rFlux2TransformerBlock.forwardzs"8!=2N]MbMbcoqrMsJ(x*J9YhYnYn !Z V.:0Vk: "ZZ 6)m/AAIM&*%7%78M%N"&'+o9S%SVa%a"!II ,"<- %  ,=( (, % 3 !ZZ 6/1y=AIMGG./ %9(<< )+>> 58K K%)%7%78M%N"%?1{?%SVa%a" OO,FG 5 EV8V V & &%-- 7$9$>$>vu$M !$33r))rryr{r@rvrwrxrzrorFrqrs@r*rtrtWs  \  \! \ \  \  \ \ \PFJ8<94||94 %||94ll 94 ll 94   ell :;dB 94!%S#X 594 u||U\\) *9494r)rtct^\rSrSrS\S\\4U4SjjrS\RS\R4Sjr Sr U=r $) Flux2PosEmbedithetaaxes_dimc:>[TU]5 XlX lgr/)rr5rr)r4rrrs r*r5Flux2PosEmbed.__init__s   r)idsr:c /n/nUR5nURRS:HnURRS:HnU(dU(a[RO[R n[ [UR55HQn[URUUSU4URSSUS9upURU 5 URU 5 MS [R"USS9RUR5n [R"USS9RUR5n X4$)Nmpsnpu.T)rrepeat_interleave_realuse_real freqs_dtyperrg)rdevicetyperBfloat32float64rLr*rrrrrir) r4rcos_outsin_outposis_mpsis_npuricossin freqs_cos freqs_sins r*rFlux2PosEmbed.forwardsiikE)E)(.&emmu}} s4==)*A. a CF jj'+' HC NN3  NN3 +IIg2.11#**= IIg2.11#**= ##r))rr) r"r#r$r%r^listr5rBrCrr(rrs@r*rrs:!c!T#Y! $5<<$ELL$$r)rc ^\rSrSrS S\S\S\S\4U4SjjjrS\RS\RS \R4S jr S r U=r $) Flux2TimestepGuidanceEmbeddingsi in_channels embedding_dimrguidance_embedsc>[TU]5 [USSS9Ul[ XUS9UlU(a[ XUS9UlgSUlg)NTr) num_channelsflip_sin_to_cosdownscale_freq_shift)rtime_embed_dimsample_proj_bias)rr5r time_projrtimestep_embedderguidance_embedder)r4rrrrrs r*r5(Flux2TimestepGuidanceEmbeddings.__init__sV " Thij!2#TX"  %6'X\&D "&*D "r)timestepguidancer:cURU5nURURUR55nUbNURbAURU5nUR URUR55nXF-nU$U$r/)rrrrr)r4rrtimesteps_proj timesteps_emb guidance_proj guidance_embtime_guidance_embs r*r'Flux2TimestepGuidanceEmbeddings.forwards1..~/@/@/PQ  D$:$:$F NN84M11-2B2B8>>2RSL - < $ $ r))rrr)iFT) r"r#r$r%r^rr5rBrCrr(rrs@r*rrsl! $ *** *  *** ! ! ! ! !r)rc ^\rSrSrS S\S\S\4U4SjjjrS\RS\R4Sjr \ S \RS\S\ \ \R\R\R4S 44S j5r S r U=r$)rgirhmod_param_setsrc>[TU]5 X l[R"XS-UR-US9Ul[R "5Ulg)Nrr)rr5rrrlinearrr)r4rhrrrs r*r5Flux2Modulation.__init__sB ,ii1Wt/B/B%BN ggi r)tembr:cJURU5nURU5nU$r/)rr)r4rmods r*rFlux2Modulation.forwards#kk$kk# r)r.c^URS:XaURS5n[R"USU-SS9m[ U4Sj[ U555$)Nrrfrrrgc3@># UHnTSU-SUS--v M g7f)rrfNr!).0r mod_paramss r* (Flux2Modulation.split.. s&T>SZAQU 4>Ss)rrrBrrDrL)rrrs @r*r?Flux2Modulation.splitsK 88q=--"C[[a.&8bA TeN>STTTr))rrr)rF)r"r#r$r%r^rr5rBrCr staticmethodrDr?r(rrs@r*rgrgs C    ELLU\\ U5<<UUuU\\SXS_S_afamamEm?nps?s9tUUr)rgc(^\rSrSrSrSrSS/rSS/rSS/r\ "SS S S 9\ "SS S S 9\ "SS S S 9\ "SS S S 9S .\ "SS S 9S.r \ S0S\ S\ S\ S-S\ S\ S\ S\ S\ S\ S\S\\ S4S\ S\S\4U4Sjjj5rS /r\"S!5S1S"\R*S#\R*S$\R,S%\R*S&\R*S'\R*S!\\\4S-S(\S S)S*\S-S+\ S,\S-\R*\-4S.jj5rS/rU=r$)2Flux2Transformer2DModelia The Transformer model introduced in Flux 2. Reference: https://blackforestlabs.ai/announcing-black-forest-labs/ Args: patch_size (`int`, defaults to `1`): Patch size to turn the input data into small patches. in_channels (`int`, defaults to `128`): The number of channels in the input. out_channels (`int`, *optional*, defaults to `None`): The number of channels in the output. If not specified, it defaults to `in_channels`. num_layers (`int`, defaults to `8`): The number of layers of dual stream DiT blocks to use. num_single_layers (`int`, defaults to `48`): The number of layers of single stream DiT blocks to use. attention_head_dim (`int`, defaults to `128`): The number of dimensions to use for each attention head. num_attention_heads (`int`, defaults to `48`): The number of attention heads to use. joint_attention_dim (`int`, defaults to `15360`): The number of dimensions to use for the joint attention (embedding/channel dimension of `encoder_hidden_states`). pooled_projection_dim (`int`, defaults to `768`): The number of dimensions to use for the pooled projection. guidance_embeds (`bool`, defaults to `True`): Whether to use guidance embeddings for guidance-distilled variant of the model. axes_dims_rope (`tuple[int]`, defaults to `(32, 32, 32, 32)`): The dimensions to use for the rotary positional embeddings. TrtrW pos_embedr]rfrF) split_dim expected_dims split_output)rrimg_idstxt_ids) gather_dimr)proj_outN patch_sizer out_channels num_layersrIrYrXjoint_attention_dimtimestep_guidance_channelsrPaxes_dims_rope. rope_thetarrc>[TU]5 U=(d UUlXv-Ul[ XS9Ul[ U URSUS9Ul[URSSS9Ul [URSSS9Ul [URSSS9Ul [R"X RSS9Ul[R"XRSS9Ul[R "[#U5Vs/sHn[%URUUU U SS9PM sn5Ul[R "[#U5Vs/sHn[)URUUU U SS9PM sn5Ul[-URURSU SS 9Ul[R"URX-UR-SS9UlSUlgs snfs snf) N)rrF)rrrrr)rrrfr)rhrXrYrPrr)rrr)rr5rrrrrtime_guidance_embedrgdouble_stream_modulation_imgdouble_stream_modulation_txtsingle_stream_modulationrr x_embeddercontext_embedderrrLrttransformer_blocksrWsingle_transformer_blocksrnorm_outrgradient_checkpointing)r4rrrrrIrYrXrrrPrrrrrPrs r*r5 Flux2Transformer2DModel.__init__Fs$ (7K,A'ZQ$C2..+ $  -H''; $ A A$ G $ A A$ G 99$? I %.1*<*003N#"%d&=&=">"%d&D&D"EH'5 #!??8$5NOL// hGH!%!B!B8!L !::8DN%=%'9>>% !  6 $ 5 56K L <<1 ajG <<1 ajG>>'2..1 IIq)+;A+>?Q G IIq)+;A+>?Q G  I %)/R !*"0 N h &8+?)/R !)"*"9"9 N4N#,D,C,C"D K(X-A-5-@-@-Mz*$$&&4+F+F7;7X7X!)))%"84%}8="/*?!6!6%6+9 84%}#E2 #8"HaP  I %.1*<,2215  8!>>Sc!   $$X~$X7G$X !$2 !#,D,J,J"K K(X-A4<4G4G 4T%j1$$&&4+F+F $ A A!%%) ! !&"/*..%6+@ ! #L. I %.1*<)!^-L-NPS*STM)!^_c*ABM m: }- I %))0R R9 ,F;;r))rrrrrrrrrrrrrr)rfNr30rri<rr) rrrirpT) NNNNNNTNNrr5)r"r#r$r%r& _supports_gradient_checkpointing_no_split_modules _skip_layerwise_casting_patterns_repeated_blocksrr_cp_planrr^rrDrr5 _skip_keysr rBrC LongTensorrrr rrrr(rrs@r*rrsq>(,$02OP(3V'<$/1NO2AQ]bc%9AUVej%k+aqW\]+aqW\]  *QaHH#'!#"%#%#(*-*: $O,O,O,Dj O,  O,  O, O,!O,!O,%(O,O,c3hO,O,O,O,O,bJ.//3%) $ $!%8< *.$($'J<||J< %||J<"" J<  J<  J<,,J<!%S#X 5J<J<(J<TzJ<J<"J< 5 5J<0Jr*s! B^^::L<6$ (2   H %  +J +  +6  >*. 9< <<9< 9< <<9< 9<  9<  $& 9< \\9