3j SSKrSSKJr SSKrSSKrSSKJr SSKJs J r SSK J r J r SSKJrJrJr SSKJrJr SSKJr SS KJrJr SS KJrJrJr SS KJr SS K J!r! SS K"J#r#J$r$J%r%J&r& SSK'J(r( SSK)J*r* SSK+J,r,J-r-J.r. \R^"\05r1S"S#Sjjr2S"S#Sjjr3S"S#Sjjr4"SS5r5"SS\RRl5r7"SS\RRl\5r8\"SS\Rl55r9\"SS\Rl55r:"SS\Rl5r;"S S!\*\ \\\\!\5 r oo&;< (=> u[ GGcURU5RSSS9up4nS=n=pxUb3[US5(a"URU5RSSS9upgnX4XVXx4$)Nrdimr to_added_qkv)to_qkvchunkhasattrr9r(s r2_get_fused_projectionsr=;sy M2888CE299M9K(WT>-J-J484E4EF[4\4b4bcdjl4b4m1 M u[ GGr4cTUR(a [XU5$[XU5$r )fused_projectionsr=r3)r)r*r+s r2_get_qkv_projectionsr@Es' %d;PQQ D1F GGr4c \rSrSrSrSrSrS SSS\RS\RS\RS-S \RS-S \R4 S jjr S r g)FluxAttnProcessorKNcr[[S5(d"[URRS35eg)Nscaled_dot_product_attentionz; requires PyTorch 2.0. Please upgrade your pytorch version.)r<F ImportError __class____name__)selfs r2__init__FluxAttnProcessor.__init__Os3q899!8!8 99tuv v:r4r)rr*r+attention_maskimage_rotary_embreturnc [XU5upgppURSURS45nURSURS45nURSURS45nURU5nUR U5nUR bU RSURS45n U RSURS45n U RSURS45n UR U 5n URU 5n [R"X/SS9n[R"X/SS9n[R"X/SS9nUb[XeSS9n[XuSS9n[UUUUURURS9nURSS5nURUR 5nUbUR#UR$SUR$SUR$S- /SS9up2UR&S"UR)55nUR&S"U5nUR+UR)55nX#4$U$) Nr6r7 sequence_dim) attn_maskbackendparallel_configr rr)r@ unflattenheadsnorm_qnorm_kr$ norm_added_q norm_added_ktorchcatrr_attention_backend_parallel_configflattentodtypesplit_with_sizesshapeto_out contiguous to_add_out) rJr)r*r+rMrNr,r-r.r/r0r1s r2__call__FluxAttnProcessor.__call__SsUH\ !6H DE+TZZ$45mmBR 01TZZ$45 E"kk#  ! ! -)33BR8HIM%//TZZ4DEK)33BR8HIM --m[TU]5 [[S5(d"[ UR R S35eXlX l[U[[45(dU/n[U[5(dU/[U5-n[U5[U5:wa [S5eX@l[R "[#[U55Vs/sHn[R$"X!SXVS9PM sn5Ul[R "[#[U55Vs/sHn[R$"X!SXVS9PM sn5Ulgs snfs snf)NrEz@ requires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0.zJ`scale` should be a list of integers with the same length as `num_tokens`.T)biasdevicerc)superrKr<rFrGrHrIrurv isinstancetuplelistlen ValueErrorscalenn ModuleListrangeLinearto_k_ipto_v_ip) rJrurv num_tokensrryrc_rHs r2rK#FluxIPAdapterAttnProcessor.__init__s9 q899>>**++kl '#6 *udm44$J%&&Gc*o-E u:Z (ij j }}s:/ /A -fb/   }}s:/ /A -fb/      s $ E#6 E(r)rr*r+rMrNip_hidden_statesip_adapter_masksrOc URSn[XU5upppU RSURS45n U RSURS45n U RSURS45n UR U 5n UR U 5n U nUbU RSURS45n U RSURS45n URSURS45nUR U 5n URU 5n [R"X/SS9n [R"X/SS9n [R"X/SS9n Ub[XSS9n [XSS9n [U U U USSURURS9nURS S 5nURU R 5nUGbUR#URSURSURS- /SS9up2UR$S"U5nUR$S"U5nUR'U5n[R("U5n[+X`R,UR.UR05HunnnnU"U5nU"U5nUR3USURUR45nUR3USURUR45n[UUUSSSURURS9nUR7USURUR4-5nURUR 5nUUU-- nM X#U4$U$) Nrr6rQr7rRF)rT dropout_p is_causalrUrVr r)rer@rWrXrYrZr[r\r]r^rrr_r`rarbrcrdrfrh zeros_likeziprrrviewhead_dimreshape)rJr)r*r+rMrNrr batch_sizer,r-r.r/r0r1ip_queryip_attn_outputcurrent_ip_hidden_statesrrrip_keyip_values r2ri#FluxIPAdapterAttnProcessor.__call__sj#((+ G[ !6H DE+TZZ$45mmBR 01TZZ$45 E"kk# ,)33BR8HIM%//TZZ4DEK)33BR8HIM --m[TU]5 X0lU bU OX2-UlXlXPlX@lU bU OUUlXlXl U bX-OUUl X`l Xpl [RRX9U S9Ul[RRX9U S9Ul[RR%XRUS9Ul[RR%XRUS9Ul[RR%XRUS9UlUR(d[RR-/5UlUR.R1[RR%URURUS95 UR.R1[RR3U55 Ub[RRX9S9Ul[RRX9S9Ul[RR%X`RUS9Ul[RR%X`RUS9Ul[RR%X`RUS9Ul[RR%URXS9UlUcURA5nURCU5 g)N)rrrx)r)"rzrKr inner_dimruse_biasrrrrrXr$rr]rRMSNormrYrZrr!r"r#rrfappendDropoutr[r\r%r&r'rh_default_processor_cls set_processor)rJrrXrrrxr$rrrrrrr processorrHs r2rKFluxAttention.__init__s5"  $+$7X=M"  ")"5w9 0 ,3,?W(U !2.hh&&xM_&` hh&&xM_&` HHOOI~~DOI HHOOI~~DOI HHOOI~~DOI }}((--b1DK KK  uxxt~~t||RZ[ \ KK  uxx//8 9  ( % 0 0 0 CD  % 0 0 0 CD #hhoo.?VeofDO#hhoo.?VeofDO#hhoo.?VeofDO#hhoodnnioWDO  335I 9%r4r*r+rMrNrOc .[[R"URR5R R 55nSS1nUR5VV s/sHupX;dM X;dMUPM n nn [U 5S:a:[RSU SURRRS35 UR5VV s0sHupX;dM X_M nnn UR"XX#U40UD6$s sn nfs sn nf)Nrrrzjoint_attention_kwargs z are not expected by z and will be ignored.) setinspect signaturerri parameterskeysitemsr~loggerwarningrHrI) rJr*r+rMrNkwargsattn_parametersquiet_attn_parameterskr unused_kwargsws r2forwardFluxAttention.forwardPsg//0G0GHSSXXZ[!35G H'-||~u~tq9QVWVt~ u }  ! NN)-8MdnnNfNfNoNoMpqFG $*<<>J>41Q5I$!$>J~~d3HZjuntuu v Ks# D 2D 9D  D-D)r&r%r'r$rrrrrXrr\r[rZrYrrrrhr"rfr!r#r) @rFNTTgh㈵>NNFTNrl)rIrmrnrorBrrs_available_processorsrfloatboolrKr]rprrqrrs@r2rrsF."(,'+(,#'4&4&4& 4&  4&  4&:4&4&4&4&4&+4&4&!4&4&r6:.204 v||v %||d2v t+ v  ,,- v vvr4c ^\rSrSrSS\S\S\S\4U4SjjjrSS\RS \RS \RS \ \R\R4S-S \ \ \ 4S-S \ \R\R44 Sjjr SrU=r$)FluxSingleTransformerBlockicr8num_attention_headsattention_head_dim mlp_ratioc l>[TU]5 [X-5Ul[ U5Ul[ R"XR5Ul[ R"SS9Ul [ R"XR-U5Ul [UUUUS[5SSS9Ulg)Ntanh) approximateTư>)rrrXrrxrrr)rzrKrmlp_hidden_dimrnormrrproj_mlpGELUact_mlpproj_outrrBr))rJr8rrrrHs r2rK#FluxSingleTransformerBlock.__init__es !#/2*3/  #':':; ww62  #(;(;";SA !'%')   r4Nr*r+tembrNjoint_attention_kwargsrOcURSn[R"X!/SS9nUnURXS9upUR UR U55n U=(d 0nUR "SUUS.UD6n [R"X/SS9nU RS5n XRU5-nXq-nUR[R:XaURSS5nUSS2SU24USS2US24pX!4$) NrQr7emb)r*rNr rk) rer]r^rrrr) unsqueezerrcfloat16clip) rJr*r+rrNr text_seq_lenresidualnorm_hidden_statesgatemlp_hidden_states attn_outputs r2r"FluxSingleTransformerBlock.forwardys-2215  #8"HaP  #'99]9#E  LL7I)JK!7!=2ii ,- %  ;"BJ ~~a }}];;  0   %-- /)..vu=M/[TU]5 [U5Ul[U5Ul[ UUUUUSS[ 5US9 Ul[R"USSS9Ul [XSS9Ul [R"USSS9Ul [XSS9Ulg)NFT) rr$rrXrrrxrrrrrzgelu-approximate)r8dim_out activation_fn)rzrKrnorm1 norm1_contextrrBr)r LayerNormnorm2rff norm2_context ff_context)rJr8rrrrrHs r2rKFluxTransformerBlock.__init__s %c* -c2!!'%"')   \\#%TJ #BTU\\#%TR%#J\]r4Nr*r+rrNrrOc URXS9upgpn URX#S9uppnU=(d 0nUR"SUU US.UD6n[U5S:XaUunnO[U5S:XaUunnnUR S5W-nUU-nUR U5nUSU SS2S4--USS2S4-nUR U5nU R S5U-nUU-n[U5S:XaUW-nU R S5W-nUU-nURU5n U SUSS2S4--U SS2S4-n URU 5nX/R S5U--nUR[R:XaURSS5nX!4$) Nr)r*r+rNr rrQrrrk) rrr)r~rrrrrrcr]rr)rJr*r+rrNrrgate_msa shift_mlp scale_mlpgate_mlpnorm_encoder_hidden_states c_gate_msa c_shift_mlp c_scale_mlp c_gate_mlpattention_outputsrcontext_attn_outputr ff_outputcontext_ff_outputs r2rFluxTransformerBlock.forwards HLzzR_zGjDiHW[WiWi !XjX T" *"8!=2!II ,"<- %    !Q &/@ ,K, " #q (?P $>vu$M !$33r4)r)rrrrrr)rms_normrr)rIrmrnrorrrrKr]rpr|rrrrqrrs@r2rrstx^^-0^FI^TW^kp^^>FJ8< 64||64 %||64ll 64   ell :;dB 64 !%S#X 5 64 u||U\\) *6464r4rct^\rSrSrS\S\\4U4SjjrS\RS\R4Sjr Sr U=r $) FluxPosEmbedithetaaxes_dimc:>[TU]5 XlX lgr )rzrKrr)rJrrrHs r2rKFluxPosEmbed.__init__s   r4idsrOc URSn/n/nUR5nURRS:HnURRS:HnU(dU(a[R O[R n[U5HSn [URU USS2U 4URSSUS9upURU 5 URU 5 MU [R"USS9RUR5n [R"USS9RUR5n X4$)Nr6mpsnpuT)rrepeat_interleave_realuse_real freqs_dtyper7)rerrytyper]float32float64rrrrrr^rb)rJrn_axescos_outsin_outposis_mpsis_npuricossin freqs_cos freqs_sins r2rFluxPosEmbed.forwards2iikE)E)(.&emmu}} vA. a AqD jj'+' HC NN3  NN3 IIg2.11#**= IIg2.11#**= ##r4)rr) rIrmrnrorr}rKr]rprrqrrs@r2rrs:!c!T#Y! $5<<$ELL$$r4rc^\rSrSrSrSrSS/rSS/rSS/r\ "SS S S 9\ "SS S S 9\ "S S S S 9\ "S S S S 9S.\ "SS S9S.r \ S+S\ S\ S\ S-S\ S\ S\ S\ S\ S\ S\S\\ \ \ 44U4Sjjj5r\"S5S,S\R&S \R&S!\R&S"\R(S#\R&S$\R&S%\R&S\\\4S-S&\S'\S(\R&\-4S)jj5rS*rU=r$)-FluxTransformer2DModeli a The Transformer model introduced in Flux. Reference: https://blackforestlabs.ai/announcing-black-forest-labs/ Args: patch_size (`int`, defaults to `1`): Patch size to turn the input data into small patches. in_channels (`int`, defaults to `64`): The number of channels in the input. out_channels (`int`, *optional*, defaults to `None`): The number of channels in the output. If not specified, it defaults to `in_channels`. num_layers (`int`, defaults to `19`): The number of layers of dual stream DiT blocks to use. num_single_layers (`int`, defaults to `38`): The number of layers of single stream DiT blocks to use. attention_head_dim (`int`, defaults to `128`): The number of dimensions to use for each attention head. num_attention_heads (`int`, defaults to `24`): The number of attention heads to use. joint_attention_dim (`int`, defaults to `4096`): The number of dimensions to use for the joint attention (embedding/channel dimension of `encoder_hidden_states`). pooled_projection_dim (`int`, defaults to `768`): The number of dimensions to use for the pooled projection. guidance_embeds (`bool`, defaults to `False`): Whether to use guidance embeddings for guidance-distilled variant of the model. axes_dims_rope (`tuple[int]`, defaults to `(16, 56, 56)`): The dimensions to use for the rotary positional embeddings. Trr pos_embedrrQrF) split_dim expected_dims split_outputrr )r*r+img_idstxt_ids) gather_dimr0)rN patch_size in_channels out_channels num_layersnum_single_layersrrjoint_attention_dimpooled_projection_dimguidance_embedsaxes_dims_ropec L>[TU]5 U=(d UUlXv-Ul[ SU S9UlU (a[ O[n U "URU S9Ul[R"XR5Ul [R"X R5Ul [R"[U5V s/sHn [URUUS9PM sn 5Ul[R"[U5V s/sHn [#URUUS9PM sn 5Ul['URURSSS9Ul[R"URX-UR-SS 9UlSUlgs sn fs sn f) Ni')rr) embedding_dimr<)r8rrFrrTr)rzrKr8rrr.rrtime_text_embedrrcontext_embedder x_embedderrrrtransformer_blocksrsingle_transformer_blocksrnorm_outrgradient_checkpointing)rJr6r7r8r9r:rrr;r<r=r>text_time_guidance_clsrrHs r2rKFluxTransformer2DModel.__init__Csw (7K,A%ENK;J 6Oq  6..@U !# *=~~ N))K@"$--z*   +A %(;'9 +   # *,01   2A +(;'9 2   * &/t~~t~~bgmqr  $..*2IDL]L]2]dhi &+#1  s FF!rr*r+pooled_projectionstimestepr2r3guidance return_dictcontrolnet_blocks_repeatrOc URU5nURUR5S-nUbURUR5S-nUcURXC5OURXGU5n UR U5nUR S:Xa[ RS5 USnUR S:Xa[ RS5 USn[R"Xe4SS9nURU5nUb;SU;a5URS5nURU5nURSU05 [UR5Hunn[R "5(a*UR"(aUR%UUUU UU5up!O U"UUU UUS 9up!U cM[['UR5['U 5- n[)[*R,"U55nU (aXU['U 5--nMXUU--nM [UR.5Hunn[R "5(a*UR"(aUR%UUUU UU5up!O U"UUU UUS 9up!U cM[['UR.5['U 5- n[)[*R,"U55nXUU--nM UR1X5nUR3U5nU (dU4$[5US 9$) a The [`FluxTransformer2DModel`] forward method. Args: hidden_states (`torch.Tensor` of shape `(batch_size, image_sequence_length, in_channels)`): Input `hidden_states`. encoder_hidden_states (`torch.Tensor` of shape `(batch_size, text_sequence_length, joint_attention_dim)`): Conditional embeddings (embeddings computed from the input conditions such as prompts) to use. pooled_projections (`torch.Tensor` of shape `(batch_size, projection_dim)`): Embeddings projected from the embeddings of input conditions. timestep ( `torch.LongTensor`): Used to indicate denoising step. block_controlnet_hidden_states: (`list` of `torch.Tensor`): A list of tensors that if specified are added to the residuals of transformer blocks. joint_attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under `self.processor` in [diffusers.models.attention_processor](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py). return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~models.transformer_2d.Transformer2DModelOutput`] instead of a plain tuple. Returns: If `return_dict` is True, an [`~models.transformer_2d.Transformer2DModelOutput`] is returned, otherwise a `tuple` where the first element is the sample tensor. irzrPassing `txt_ids` 3d torch.Tensor is deprecated.Please remove the batch dimension and pass it as a 2d torch TensorrzrPassing `img_ids` 3d torch.Tensor is deprecated.Please remove the batch dimension and pass it as a 2d torch Tensorr7ip_adapter_image_embedsr)r*r+rrNr)sample)rCrbrcrArBndimrrr]r^r.popencoder_hid_projupdate enumeraterDis_grad_enabledrG_gradient_checkpointing_funcr~rnpceilrErFrr)rJr*r+rJrKr2r3rLrcontrolnet_block_samplescontrolnet_single_block_samplesrMrNrrrNrPr index_blockblockinterval_controloutputs r2rFluxTransformer2DModel.forward}s+V 6 ;;}223d:  {{=#6#67$>H   >%%h:LM !% 5 56K L <<1  NNU ajG <<1  NNU ajGii*2>>#. ! -2KOe2e&<&@&@AZ&[ ##445LM  " ) )+=?O*P Q"+D,C,C"D K$$&&4+F+F7;7X7X!)$* 84%}8="/*?%5+A 84%(3#&t'>'>#?#F^B_#_ #&rww/?'@#A +%sSkOlAl(mm"%2[\lMl4m$mM=#E@#,D,J,J"K K$$&&4+F+F7;7X7X!)$* 84%}8="/*?%5+A 84%/:#&t'E'E#FMlIm#m #&rww/?'@#A -P[_oPo0p p 1#L4 m: }-9 'v66r4) rBrGrrFr8r.rrErArDrC) rQrN&iiF)8rg) NNNNNNNNNTF)rIrmrnror _supports_gradient_checkpointing_no_split_modules _skip_layerwise_casting_patterns_repeated_blocksrr_cp_planrrrr|rKr r]rp LongTensorrrrrrrqrrs@r2r-r- s >(,$/1MN(3V'<$.0LM2AQ]bc%9AUVej%k+aqW\]+aqW\]  *QaHH#'!#"%#%#'%( %/;7,7,7,Dj 7,  7,  7, 7,!7,!7, #7,7,c3m,7,7,r.//3+/%) $ $!%8rs* B^^.5LII6$ 8'\\   H % HHH @!@!FB!B!JMvEHHOO%9Mv`242424jQ4299Q4Q4h$299$>}7!}7r4