3jAlSSKJr SSKJr SSKJr SSKrSSKJrJr SSK J r J r SSK J r Jr SS KJr S S KJr S S KJrJrJrJrJrJr S S KJrJr S SKJr S SKJ r J!r!J"r"J#r#J$r$J%r%J&r& S SK'J(r( SSK)J*r* \RV"\,5r-\"SS\ 55r."SS\R^5r0"SS\R^5r1"SS\R^5r2SS>S9jr?S:r@S;rAg)?) dataclass)gcd)AnyN)Tensornn) ConfigMixinregister_to_config) BaseOutputlogging) apply_freeu)AttentionMixin)ADDED_KV_ATTENTION_PROCESSORSCROSS_ATTENTION_PROCESSORS AttentionAttnAddedKVProcessor AttnProcessorFusedAttnProcessor2_0)TimestepEmbedding Timesteps) ModelMixin)CrossAttnDownBlock2DCrossAttnUpBlock2D Downsample2D ResnetBlock2DTransformer2DModelUNetMidBlock2DCrossAttn Upsample2D)UNet2DConditionModel)ControlNetConditioningEmbeddingc(\rSrSr%SrSr\\S'Srg)ControlNetXSOutput3a% The output of [`UNetControlNetXSModel`]. Args: sample (`Tensor` of shape `(batch_size, num_channels, height, width)`): The output of the `UNetControlNetXSModel`. Unlike `ControlNetOutput` this is NOT to be added to the base model output, but is already the final output. Nsample) __name__ __module__ __qualname____firstlineno____doc__r&r__annotations____static_attributes__r'd/home/wildlama/miniconda3/lib/python3.13/site-packages/diffusers/models/controlnets/controlnet_xs.pyr$r$3sFFr/r$c ^\rSrSrSrS S\R S\R S\R S\R S-S\RS-4 U4S jjjrS r U=r $) DownBlockControlNetXSAdapterAzyComponents that together with corresponding components from the base model will form a `ControlNetXSCrossAttnDownBlock2D`Nresnets base_to_ctrl ctrl_to_base attentions downsamplerc^>[TU]5 XlX lX0lX@lXPlgN)super__init__r4r5r6r7 downsamplers)selfr4r5r6r7r8 __class__s r0r<%DownBlockControlNetXSAdapter.__init__Es,  (($'r/)r7r5r6r=r4)NN) r(r)r*r+r,r ModuleListConv2dr<r. __classcell__r?s@r0r2r2Ash*,0(, ( (mm (mm ( MMD( ( YY% ( (r/r2cd^\rSrSrSrS\S\RS\R4U4SjjrSr U=r $)MidBlockControlNetXSAdapterUzxComponents that together with corresponding components from the base model will form a `ControlNetXSCrossAttnMidBlock2D`midblockr5r6cF>[TU]5 XlX lX0lgr:)r;r<rHr5r6)r>rHr5r6r?s r0r<$MidBlockControlNetXSAdapter.__init__Ys  ((r/)r5r6rH) r(r)r*r+r,rrrAr<r.rCrDs@r0rFrFUs3))!8) )egerer))r/rFcH^\rSrSrSrS\R 4U4SjjrSrU=r $)UpBlockControlNetXSAdapter`zwComponents that together with corresponding components from the base model will form a `ControlNetXSCrossAttnUpBlock2D`r6c.>[TU]5 Xlgr:)r;r<r6)r>r6r?s r0r<#UpBlockControlNetXSAdapter.__init__cs (r/r6) r(r)r*r+r,rrAr<r.rCrDs@r0rLrL`sB)R]]))r/rLbase_in_channelsbase_out_channelsctrl_in_channelsctrl_out_channels temb_channelsmax_norm_num_groupstransformer_layers_per_blocknum_attention_headscross_attention_dimadd_downsampleupcast_attentionuse_linear_projectionc FSn /n/n/n/n[U[5(aU/U -n[U 5HnUS:XaUOUnUS:XaUOUnUR[ X55 UR[ X -UU[ X -US9[ X5S9SS95 U(a+UR[UX8-UUUU U U [ X5S9S95 UR[ X155 M U (aCUR[ X55 [X1-SUSS 9nUR[ X155 OSn[[R"U5[R"U5[R"U5S 9nU(a[R"U5Ul UbUUl U$) Nrr max_factorh㈵> in_channels out_channelsrUgroups groups_outepsrb num_layersrYr\r[norm_num_groupsTopuse_convrcname)r4r5r6) isinstanceintrangeappendmake_zero_convrfind_largest_factorrrr2rrAr7r=)rQrRrSrTrUrV has_crossattnrWrXrYrZr[r\rhr4r7r6r5ir=down_block_componentss r0get_down_block_adapterrwhsJGJLL.44(D'E 'R$ : /0Av+;L/0Av+;L N+;NO ,?.+*+;+N[no./@a       "'%< 1;A>(;*?%5$78I$j   N+<PQGJ N+<PQ#  1DO`gk N+<PQ 8 g&]]<0]]<0 +-==+D(-9*  r/ base_channels ctrl_channelsc [X5n [UX-UU[[XU-5U5UUUUS9 n [X5n [ XU S9$)N rWrbrcrU resnet_groupsrYrXr\r[)r5rHr6)rrrrsrrF) rxryrUrVrWrXrYr[r\r5rHr6s r0get_mid_block_adapterr}sb"-?L&%A!1"#)#m]=Z*[]pq//3) H"-?L &Lbn oor/rcprev_output_channelctrl_skip_channelsc/nSn[U5H*nUS:XaUOUnUR[X%U55 M, [[R "U5S9$)NrrrP)rprqrrrLrrA)rcr~rr6rhruresnet_in_channelss r0get_up_block_adapterrs\ LJ : 45F0 N+=+@BTUV &2==3N OOr/c D^\rSrSrSr\SS\S\S\\S\ S\ S\\\-S \\S \\S \S \\S \S-S\\\-S\ S\S\ 4U4Sjjj5r \ SS\ S\ S-S \\S-S\\S-S\ S\S\S\S\\4Sjj5rSrSrU=r$)ControlNetXSAdaptera A `ControlNetXSAdapter` model. To use it, pass it into a `UNetControlNetXSModel` (together with a `UNet2DConditionModel` base model). This model inherits from [`ModelMixin`] and [`ConfigMixin`]. Check the superclass documentation for it's generic methods implemented for all models (such as downloading or saving). Like `UNetControlNetXSModel`, `ControlNetXSAdapter` is compatible with StableDiffusion and StableDiffusion-XL. It's default parameters are compatible with StableDiffusion. Parameters: conditioning_channels (`int`, defaults to 3): Number of channels of conditioning input (e.g. an image) conditioning_channel_order (`str`, defaults to `"rgb"`): The channel order of conditional image. Will convert to `rgb` if it's `bgr`. conditioning_embedding_out_channels (`tuple[int]`, defaults to `(16, 32, 96, 256)`): The tuple of output channels for each block in the `controlnet_cond_embedding` layer. time_embedding_mix (`float`, defaults to 1.0): If 0, then only the control adapters's time embedding is used. If 1, then only the base unet's time embedding is used. Otherwise, both are combined. learn_time_embedding (`bool`, defaults to `False`): Whether a time embedding should be learned. If yes, `UNetControlNetXSModel` will combine the time embeddings of the base model and the control adapter. If no, `UNetControlNetXSModel` will use the base model's time embedding. num_attention_heads (`list[int]`, defaults to `[4]`): The number of attention heads. block_out_channels (`list[int]`, defaults to `[4, 8, 16, 16]`): The tuple of output channels for each block. base_block_out_channels (`list[int]`, defaults to `[320, 640, 1280, 1280]`): The tuple of output channels for each block in the base unet. cross_attention_dim (`int`, defaults to 1024): The dimension of the cross attention features. down_block_types (`list[str]`, defaults to `["CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "DownBlock2D"]`): The tuple of downsample blocks to use. sample_size (`int`, defaults to 96): Height and width of input/output sample. transformer_layers_per_block (`int | tuple[int]`, defaults to 1): The number of transformer blocks of type [`~models.attention.BasicTransformerBlock`]. Only relevant for [`~models.unet_2d_blocks.CrossAttnDownBlock2D`], [`~models.unet_2d_blocks.UNetMidBlock2DCrossAttn`]. upcast_attention (`bool`, defaults to `True`): Whether the attention computation should always be upcasted. max_norm_num_groups (`int`, defaults to 32): Maximum number of groups in group normal. The actual number will be the largest divisor of the respective channels, that is <= max_norm_num_groups. conditioning_channelsconditioning_channel_order#conditioning_embedding_out_channelstime_embedding_mixlearn_time_embeddingrXblock_out_channelsbase_block_out_channelsrYdown_block_types sample_sizeNrWr[rVr\c2>[T!U]5 USnUSS-nUS;a[SU35e[U5[U 5:wa[SUSU S35e[ U [ [ 45(dU /[U 5-n [ U [ [ 45(dU /[U 5-n [ U[ [ 45(dU/[U 5-n[U5[U 5:wa[SUSU S35e[USUUS 9UlU(a[UU5Ul OSUl [R"/5Ul [R"/5Ul[R"SUSS S S 9Ul[#USUS5UlUSnUSn['U 5HfunnUnUUnUnUUnS U;nU[U 5S - :HnURR)[+UUUUUUUU UUUU UU(+U US9 5 Mh [-USUSUU SUSU SU US9UlUS/n['U5H1unnU[U5S - :aS OSnUR1U/U-5 M3 [ [3U55nUSn[5[U 55HWnUnUUn[5S 5Vs/sHnUR75PM n nURR)[9UUU S95 MY gs snf)Nr)rgbbgrz&unknown `conditioning_channel_order`: zbMust provide the same number of `block_out_channels` as `down_block_types`. `block_out_channels`: z. `down_block_types`: .zdMust provide the same number of `num_attention_heads` as `down_block_types`. `num_attention_heads`: conditioning_embedding_channelsrrrr! kernel_sizepadding CrossAttn) rQrRrSrTrUrVrtrWrXrYrZr[r\)rxryrUrWrXrYr[r\r)rcr~r)r;r< ValueErrorlenrnlisttupler"controlnet_cond_embeddingrtime_embeddingrrA down_blocksup_connectionsrBconv_inrrcontrol_to_base_for_conv_in enumeraterqrwr} mid_blockextendreversedrppopr)"r>rrrrrrXrrrYrrrWr[rVr\time_embedding_input_dimtime_embedding_dimrRrTrudown_block_typerQrSrtis_final_blockrrcnumber_of_subblocks reversed_base_block_out_channelsprev_base_output_channel_ctrl_skip_channels_r?s" r0r<ControlNetXSAdapter.__init__#s0 #:1#= 4Q7!; &^ ;EF`Eabc c ! "c*:&; ;tvHuII_`p_qqrs 6u FF,H+ICP`La+a (-e}==#6"7#>N:O"O -e}==#6"7#>N:O"O  " #s+;'< <vxKwLLbcsbttuv  *I,>q,AB"7* & "34LN`"aD "&D ==, mmB/yy$6q$9qRST +9:LQ:OQhijQk+l(4A6.q1"+,<"= A0  7 : 0  21 5 '?:M#&6"7!";;N    # #&%5&7%5&7"4(;"/1Ma1P(;A(>(;A(>'5#5%5*? #>6/1"5,R0,)Eb)I 3B 7 3B 7-"7  134();argskwargss r0forwardControlNetXSAdapter.forwards Q  r/)rrrrrrr)rr rM?Frrrri@irrrr DownBlock2DrMr!TrT)NNNFrrrr)r(r)r*r+r,r rostrrfloatboolr< classmethodr rrrr.rCrDs@r0rrs,\&'*/:K$'%*01)7.D#'( #%9:!%#%&*+F"F%(F.33Z F " F # F!5:-F"#JF"'sF!F *F"4Z#F$'*E#J&6%F&'F(!)F* $+FFP$(/304%*"%%&*/:KJ"JDLJ!I, J "#Y- J # J J #J%(J.33ZJJX  r/rc.^\rSrSrSrSr\S:S\S-S\\ S\\ S\\S \S-S \\\-S \\\-S \\\-S \ S-S\S-S\ S\ S\S-S\S-S\ S\S\\S\ S\ S\\S\\\-S\4,U4Sjjj5r \ S;S\S\S-S\ S-S\\ S-S\ S-S\S-4 S jj5rSUNetControlNetXSModelia A UNet fused with a ControlNet-XS adapter model This model inherits from [`ModelMixin`] and [`ConfigMixin`]. Check the superclass documentation for it's generic methods implemented for all models (such as downloading or saving). `UNetControlNetXSModel` is compatible with StableDiffusion and StableDiffusion-XL. It's default parameters are compatible with StableDiffusion. It's parameters are either passed to the underlying `UNet2DConditionModel` or used exactly like in `ControlNetXSAdapter` . See their documentation for details. TNrrup_block_typesrrirYrWrXaddition_embed_typeaddition_time_embed_dimr[r\time_cond_proj_dim%projection_class_embeddings_input_dimrctrl_conditioning_channels(ctrl_conditioning_embedding_out_channelsctrl_conditioning_channel_orderctrl_learn_time_embeddingctrl_block_out_channelsctrl_num_attention_headsctrl_max_norm_num_groupsc>[T0U]5 US:dUS:a [S5eUS:aU(d [S5eU bU S:wa [S5e[U[[ 45(dU/[ U5-n[U[[ 45(dU/[ U5-n[U[[ 45(dU/[ U5-n[U[[ 45(dU/[ U5-nUnSUl[R"SUSSSS 9Ul [USUUS 9Ul [R"SUSSSS 9Ul [USUS5UlUSnUSS-n[!USS SS 9Ul[%UUU S 9UlU(a[%UUS9UlOSUlU cSUlSUlO![!U S SS 9Ul[%UU5Ul/nUSnUSn[/U5HaunnUnUUnUn UUnSU;n!U[ U5S- :Hn"UR1[3UUU UUUUU!UUUUUUUUU"(+U U S95 Mc [5USUSUUUUSUSUSUSU U S9 Ul/n#[ [9U55n$[ [9U55n%[ [9U55n&US/n'[/U5H1unn(U[ U5S- :aSOSn)U'R;U(/U)-5 M3 [ [9U55n*U*Sn([/U5Hunn+U(n,U*Un(U*[=US-[ U5S- 5n-[?S5V.s/sHn.U'RA5PM n/n.SU+;n!U[ U5S- :Hn"U#R1[CU-U(U,U/UUU!U$UU%UU&UU"(+U UU S95 M [RD"U5Ul#[RD"U#5Ul$[RJ"USUS9Ul&[RN"5Ul([R"USSSSS 9Ul)gs sn.f)Nrr!z1`time_embedding_mix` needs to be between 0 and 1.zKTo use `time_embedding_mix` < 1, `ctrl_learn_time_embedding` must be `True` text_timezAs `UNetControlNetXSModel` currently only supports StableDiffusion and StableDiffusion-XL, `addition_embed_type` must be `None` or `'text_time'`.rrrrT)flip_sin_to_cosdownscale_freq_shift) cond_proj_dim)rbtime_embed_dimrrQrRrSrTrUrirrtrWbase_num_attention_headsrrYrZr[r\r rxryrUrirrWrrrYr[r\r)rbrcr~rrUresolution_idxrtrWrXrY add_upsampler[rir\) num_channels num_groups)*r;r<rrnrrrrbrrB base_conv_inr"r ctrl_conv_inrrrrbase_time_projrbase_time_embeddingctrl_time_embeddingbase_add_time_projbase_add_embeddingrrq ControlNetXSCrossAttnDownBlock2DControlNetXSCrossAttnMidBlock2DrrrminrprControlNetXSCrossAttnUpBlock2DrAr up_blocks GroupNormbase_conv_norm_outSiLU base_conv_act base_conv_out)1r>rrrrrirYrWrXrrr[r\rrrrrrrrrrrtime_embed_input_dimrrrRrTrurrQrSrtrr rev_transformer_layers_per_blockrev_num_attention_headsrev_cross_attention_dimrrcrreversed_block_out_channels up_block_typer~rbrrr?s1 r0r<UNetControlNetXSModel.__init__s?B   !%7!%;PQ Q  !*Cjk k  */Bk/Qd 6u FF,H+ICP`La+a (-e}==#6"7#>N:O"O -e}==#6"7#>N:O"O 2T5MBB(@'ACHXDY'Y $#6 IIa);A)>AWXY)H,CA,FG"<* & IIa)@)CQR\]^+9:QRS:TVhijVk+l( 2!4+A.2'(:1(=tjkl#4 ,$  %'80(D $(,D $  &&*D #&*D #&/0GY]tu&vD #&78]_m&nD # .q13A6"+,<"= A0  21 5 0  7 : '?:M#&6"7!";;N   0%5&7%5&7"0$3-E"/1Ma1P-Ea-H-Ea-H(;A(>'5#5%5*? #>:9,R01"5(+%=)Eb)I%=b%A%=b%A 3B 7-"7   +/9U0V+W("&x0H'I"J"&x0C'D"E6a89()@AOA|4599q   % %|n7J&J K B '+84F+G&H#215 ). 9 A}". 6q9L5c!a%EWAX[\A\6]^KEJ1X"NX#5#9#9#;X "N'=8M#&8"9A"==N   . +!-(;':"0#$"/1QRS1T(?(B(?(B%3!3%5$3*? !:8==5y1"$,,2UNetControlNetXSModel.from_unet..s'v! 'vszWhen a controlnet is passed, none of these parameters should be passed: size_ratio, ctrl_block_out_channels, time_embedding_mix, ctrl_optional_kwargs.) rrrrrirYrWrrr[r\rrrX)rrrrrrXrVctrl_r)rr conv_norm_outconv_outbase_ add_time_proj add_embeddingc3P# UHup[RX5v M g7fr:)r from_modulesrrcs r0rr6s'* E - 9 9! ? ?E$&c3P# UHup[RX5v M g7fr:)rrrs r0rr;s'( F + 7 7 = =Fr)rranyrritemsrr from_configgetattrload_state_dict state_dicthasattrrrrrrrrrAziprrrrrrrr)rrr rrrr params_for_unetkvparams_for_controlnetrmodules_from_unetmoptional_modules_from_unets r0rUNetControlNetXSModel.from_unets<  ,66"9=QJ(2M_'v!m  -1KK,=,=,?X,?DA1CW414,?X151O1O-.! =Gparam base_partspartdus r0freeze_unet_params(UNetControlNetXSModel.freeze_unet_paramsEs__&E"&E '  7AdjdGDDW)gd)j dD*&+#+!!A "" ))+A " es CCc ~[SURR555(a [5nOr[SURR555(a [ 5nO8[ S[ [URR55535eURU5 g)zU Disables custom attention processors and sets the default attention implementation. c3F# UHoR[;v M g7fr:)r?rrprocs r0rCUNetControlNetXSModel.set_default_attn_processor..gsiKh4~~!>>Kh!c3F# UHoR[;v M g7fr:)r?rr=s r0rr?ishJg$#==Jgr@zOCannot call `set_default_attn_processor` when attention processors are of type N) allattn_processorsvaluesrrrnextiterset_attn_processor)r> processors r0set_default_attn_processor0UNetControlNetXSModel.set_default_attn_processorcs i4K_K_KfKfKhi i i,.I h$J^J^JeJeJgh h h%IabfgklpmAmAmHmHmJhKcLbMN   *r/s1s2b1b2c[UR5H9upV[USU5 [USU5 [USU5 [USU5 M; g)aEnables the FreeU mechanism from https://huggingface.co/papers/2309.11497. The suffixes after the scaling factors represent the stage blocks where they are being applied. Please refer to the [official repository](https://github.com/ChenyangSi/FreeU) for combinations of values that are known to work well for different pipelines such as Stable Diffusion v1, v2, and Stable Diffusion XL. Args: s1 (`float`): Scaling factor for stage 1 to attenuate the contributions of the skip features. This is done to mitigate the "oversmoothing effect" in the enhanced denoising process. s2 (`float`): Scaling factor for stage 2 to attenuate the contributions of the skip features. This is done to mitigate the "oversmoothing effect" in the enhanced denoising process. b1 (`float`): Scaling factor for stage 1 to amplify the contributions of backbone features. b2 (`float`): Scaling factor for stage 2 to amplify the contributions of backbone features. rKrLrMrNN)rrsetattr)r>rKrLrMrNruupsample_blocks r0 enable_freeu"UNetControlNetXSModel.enable_freeussJ$"+4>>!: A ND" - ND" - ND" - ND" - ";r/c1Skn[UR5H9up#UH.n[X45(d[X4S5cM"[ X4S5 M0 M; g)zDisables the FreeU mechanism.>rMrNrKrLN)rrr$r!rP)r> freeu_keysrurQr's r0 disable_freeu#UNetControlNetXSModel.disable_freeusH- !*4>>!: A>--D1Q1]Nt4 ";r/cSUlURR5H3upS[URR 5;dM*[ S5e URUlUR5H)n[U[5(dMURSS9 M+ UR[55 g)u Enables fused QKV projections. For self-attention modules, all projection matrices (i.e., query, key, value) are fused. For cross-attention modules, key and value projection matrices are fused. > [!WARNING] > This API is 🧪 experimental. NAddedzQ`fuse_qkv_projections()` is not supported for models having added KV projections.T)fuse) original_attn_processorsrCrrr?r(rmodulesrnrfuse_projectionsrGr)r>rattn_processormodules r0fuse_qkv_projections*UNetControlNetXSModel.fuse_qkv_projectionss)-%!%!5!5!;!;!= A#n66??@@ !tuu">)-(<(<%llnF&),,''T'2%  5 78r/cVURbURUR5 gg)u^Disables the fused QKV projection if enabled. > [!WARNING] > This API is 🧪 experimental. N)r[rG)r>s r0unfuse_qkv_projections,UNetControlNetXSModel.unfuse_qkv_projectionss)  ( ( 4  # #D$A$A B 5r/r&timestepencoder_hidden_statescontrolnet_condconditioning_scale class_labels timestep_condattention_maskcross_attention_kwargsadded_cond_kwargs return_dict apply_controlc  URRS:Xa[R"US/S9nUb2SUR UR 5- S-nUR S5nUn [R"U 5(dURRS:HnURRS:Hn[U[5(a/U(dU(a[RO[RnO.U(dU(a[RO[Rn[R "U /UURS9n O7[#U R$5S :XaU SR UR5n U R'UR$S 5n UR)U 5nUR UR S 9nURR*(aSU (aLUR-UU5nUR/UU5nURR0S -nUU-USU- --nOUR/U5nSnURR2cGOURR2S :XaS U ;a[5UR6S35eU R9S 5nSU ;a[5UR6S35eU R9S5nUR;UR=55nUR?UR$S S45n[R@"UU/SS9nUR UR 5nURCU5nO#[5SURR2S35eUbUU-OUnUnU=nn//nnUREU5n URGU5nURIU5nU bUU - nU (aUURKU5U--nURMU5 URMU5 URNH7n!U!"UUUUUU UU S9unnn"n#URQU"5 URQU#5 M9 URSUUUUUU UU S9unnURTH>n$[#U$RV5n%UU%*Sn&UU%*Sn'USU%*nUSU%*nU$"UU&U'UUUU UU S9 nM@ URYU5nUR[U5nUR]U5nU (dU4$[_US9$)ap The [`ControlNetXSModel`] forward method. Args: sample (`Tensor`): The noisy input tensor. timestep (`torch.Tensor | float | int`): The number of timesteps to denoise an input. encoder_hidden_states (`torch.Tensor`): The encoder hidden states. controlnet_cond (`Tensor`): The conditional input tensor of shape `(batch_size, sequence_length, hidden_size)`. conditioning_scale (`float`, defaults to `1.0`): How much the control model affects the base model outputs. class_labels (`torch.Tensor`, *optional*, defaults to `None`): Optional class labels for conditioning. Their embeddings will be summed with the timestep embeddings. timestep_cond (`torch.Tensor`, *optional*, defaults to `None`): Additional conditional embeddings for timestep. If provided, the embeddings will be summed with the timestep_embedding passed through the `self.time_embedding` layer to obtain the final timestep embeddings. attention_mask (`torch.Tensor`, *optional*, defaults to `None`): An attention mask of shape `(batch, key_tokens)` is applied to `encoder_hidden_states`. If `1` the mask is kept, otherwise if `0` it is discarded. Mask will be converted into a bias, which adds large negative values to the attention scores corresponding to "discard" tokens. cross_attention_kwargs (`dict[str]`, *optional*, defaults to `None`): A kwargs dictionary that if specified is passed along to the `AttnProcessor`. added_cond_kwargs (`dict`): Additional conditions for the Stable Diffusion XL UNet. return_dict (`bool`, defaults to `True`): Whether or not to return a [`~models.controlnets.controlnet.ControlNetOutput`] instead of a plain tuple. apply_control (`bool`, defaults to `True`): If `False`, the input is run only through the base model. Returns: [`~models.controlnetxs.ControlNetXSOutput`] **or** `tuple`: If `return_dict` is `True`, a [`~models.controlnetxs.ControlNetXSOutput`] is returned, otherwise a tuple is returned where the first element is the sample tensor. rr!)dimsNgmpsnpu)rdevicer)rg333333?r text_embedsz has the config param `addition_embed_type` set to 'text_time' which requires the keyword argument `text_embeds` to be passed in `added_cond_kwargs`time_idsz has the config param `addition_embed_type` set to 'text_time' which requires the keyword argument `time_ids` to be passed in `added_cond_kwargs`rdimzgControlNet-XS currently only supports StableDiffusion and StableDiffusion-XL, so addition_embed_type = z is currently not supported.)hidden_states_basehidden_states_ctrltembrfrhrlrkro) hidden_statesres_hidden_states_tuple_baseres_hidden_states_tuple_ctrlr{rfrhrlrkro)r&)0rrtorchfliprr unsqueeze is_tensorrttypernrfloat32float64int32int64tensorrshapeexpandrrrrrrrr?getrflattenreshapeconcatrrrrrrqrrrrr4rrrr$)(r>r&rerfrgrhrirjrkrlrmrnro timestepsis_mpsis_npurt_emb ctrl_temb base_tembinterpolation_paramr{aug_embrurv time_embeds add_embedscembh_ctrlh_basehs_basehs_ctrl guided_hintdown residual_hb residual_hcup n_resnetsskips_hbskips_hcs( r0rUNetControlNetXSModel.forwardsp ;; 6 6% ?#jjsCO  %."3"3FLL"AAXMN+55a8N y))]]''50F]]''50F(E***0F (.& u{{ i[fmmTI  !Q &!$**6==9I$$V\\!_5 ##I. v||, ;; 0 0]00 FI00 FI"&++"@"@#"E 22Y!FYBY5ZZD++E2D ;; * * 2  [[ , , ;$55 ~~&'{|,// >K!22 ~~&'xy),,Z8H11(2B2B2DEK%--{/@/@/CR.HIK{K&@bIJ#tzz2J--j9Gyz~{F{F{Z{Zz[[wx ")!4tg~$%! r44_E ""6*""6*  " k !F d>>vFI[[[Fvv$$D7;#)#)&*#5'=-+ 8 4FFK NN; ' NN; '%%%"&1#9)'(  ..BBJJI z{+H z{+Hk z*Gk z*G$-5-5&*#5'=-+ F !&((0##F+##F+9 !00r/)rrrrrrrrrrrrrrbrr[r)rMr) UpBlock2Drrrrrrr!rNNTTNNrrrrFrrr)NNNNNr.N) NrNNNNNTT) r(r)r*r+r, _supports_gradient_checkpointingr rorrrrr<rr rrdictrr9rIrRrVr`rcrrrr$rr.rCrDs@r0rrs (,$#%( &u)?&(049:01*..2!%&*)-<@$'*+?P/4*/.<56(*=|[4Z|[ * |[c |["#J|[t|[!5:-|['*E#J&6|[!5:-|[ !4Z!|[""%t#|[$%|[& $'|[( $J)|[*03Tz+|[."/|[0%(1|[238*3|[4*-5|[6$(7|[8"'s9|[:#&c "2;|[<#&=|[|[|26#'6:+/,0t"t($.tDL t "&et!3 t "DL t#Tkttl#< + .u.%.U..259,C04+.,0-1.28<<@ "J1J1,,&,J1 %|| J1 , J1 "DL J1llT)J1||d*J1 t+J1!%S#X 5J1 U\\ 12T9J1J1J1 e #J1J1r/rcf^\rSrSrS#S\S\S\S\S\S\S\S \\\-S -S \S -S \S -S \S -S\S\S -S\S -4U4Sjjjr\S\ S\ 4Sj5r S$Sjr S%S\ S\ S\ S -S\ S -S\S -S\ S -S\\\4S -S\ S -S\S\\ \ \\ S 4\\ S 444S!jjrS"rU=r$)&rirQrRrSrTrUrirrWNrrrYrZr[r\c>[TU]5 /n/n/n/n/n/nSn[U [5(aU /U-n [ U5HnUS:XaUOUnUS:XaUOUnUR [ X55 UR [UUUUS95 UR [X1-UU[X1-US9[XGS9SS95 U(aOUR [U X*-UU UU UUUS95 UR [U XK-UU UU UU[XGS9S95 UR [ XB55 M U (aYUR [ X"55 [USUS S 9Ul [XB-SUS S 9Ul UR [ XB55 OSUl SUl [R"U5Ul[R"U5UlU(a[R"U5OS/U-UlU(a[R"U5OS/U-Ul[R"U5Ul[R"U5UlS Ulg) NrrrbrcrUrdr^r`rargTrjrkF)r;r<rnrorprqrrrrsrrbase_downsamplersctrl_downsamplersrrA base_resnets ctrl_resnetsbase_attentionsctrl_attentionsr5r6gradient_checkpointing)r>rQrRrSrTrUrirrtrWrrrYrZr[r\rrrrr6r5rhrur?s r0r<)ControlNetXSCrossAttnDownBlock2D.__init__s$      2C 8 8,H+IJ+V (z"A346/?P 346/?P    /? R S    0!2"/*      0 C!2"/.(;H` 33Dj  &&&0)E$5#?#B,?.C)9(7   &&&0)E$5#?#B,?.C)9(;.get_first_cross_attention$##A&99!<BB Br/rr7TFr)r4rbrc time_emb_proj in_featuresnorm1rr$rr7rheadsrYr[r\r=rr"r#rrrrrr5r6)rrrrrQrRrSrTrUrctrl_num_groupsrtrWrrrYr[r\rZrs r0r-ControlNetXSCrossAttnDownBlock2D.from_moduless C*11!4@@*2215BB  " "1 % 1 14D D +2215BB&..q1??KK #++A.44?? (00399DD >< 0 0 M+.~/H/H/K/^/^+_ ('@'P'V'V $'@'P'V'V $";N"K"_"_ 8HYY $2$=$=a$@$V$V !!M+/ ('+ $'+ $"& # $( !'44D@-/-/'&%4')E%=%= 3)-"7 & **>+A+A+L+L+NO **>+A+A+L+L+NO   ! ! 1 1.2K2K2V2V2X Y  ! ! 1 1.2K2K2V2V2X Y   # # 3 3N4O4OPQ4R4]4]4_ `  # # 3 3N4O4O4Z4Z4\ ] **>+F+F+Q+Q+ST **>+F+F+Q+Q+ST r/r.cUR5H nSUlM UR/n[UR[ R 5(aURUR5 URbURUR5 UH!nUR5H nSUlM M# gr0TNF) r1r2rrnrrrArqrr>r4r5r6s r0r33ControlNetXSCrossAttnDownBlock2D.freeze_base_params:s__&E"&E '''( d**BMM : :   d22 3  ! ! -   d44 5D*&+#+r/ryr{rfrzrhrkrlencoder_attention_maskro.c Ub(URSS5b[RS5 Un Un Sn Sn [[ UR UR 55n[[ URUR55n[ XURUR5GHuunnunnnnU (a[R"U U"U 5/SS9n [R"5(a$UR(aURUX5n OU"X5n UbU"U UUUUSS9Sn U (aW[R"5(a$UR(aURUX5n OU"X5n UbU"U UUUUSS9Sn U (aU U"U 5U--n X4-n X4-n GM UR bURS nURS nU (a[R"U U"U 5/SS9n UR!U 5n U (aUR#U 5n U (aU U"U 5U--n X4-n X4-n XX4$) NscaleSPassing `scale` to `cross_attention_kwargs` is deprecated. `scale` will be ignored.r'r!rwFrfrlrkrrnrr)rloggerwarningrr%rrrrr5r6rcatis_grad_enabledr_gradient_checkpointing_funcrr)r>ryr{rfrzrhrkrlrrorrbase_output_statesctrl_output_states base_blocks ctrl_blocksb_resb_attnc_resc_attnb2cc2bs r0r(ControlNetXSCrossAttnDownBlock2D.forwardKsi " -%))'48Dtu##3t00$2F2FGH 3t00$2F2FGH := d&7&79J9J; 6OUF_eVc3FCK#8a@$$&&4+F+F::5&Ov,!*?+A#1+A % ((**t/J/J!>>ufSF"60F%#.C/E'5/E$) F#f+0B"BB!3i!? !3i!? Y; \  ! ! -##B'C##B'CFCK#8a@++F3F//7#f+0B"BB!3i!? !3i!? 1EEr/) rrrr5rrrr6r) rrTr!r!r!rTFTr)NNrNNNT)r(r)r*r+rorrr<rrr2rr3rrrrrrr.rCrDs@r0rrs "(*@A/0/0*.#(--1!t,t,t, t,  t,  t,t,#&t,'*E#J&6&=t,#&*t,#&*t,!4Zt,t,+t, $d{!t,t,l?*>?Pl??B,*04,0+.(,8<04"ZF"ZFZF &} ZF #TM ZF "DL ZF ZF!%S#X 5ZF!' ZFZF vvuVS[153EE FZFZFr/rc&^\rSrSrSS\S\S\S-S\S\S\S \S-S \S-S \S-S \S \S-4U4Sjjjr\S\S\ 4Sj5r S Sjr S!S\ S\ S\ S\ S-S\ S-S\\\4S-S\ S-S\ S-S\S\\ \ 44SjjrSrU=r$)"riNrxryrUrirrWrrrYr[r\c >[T U]5 [X5Ul[ UUUUU UU U S9Ul[ UX!-UU[ [X"U-5U5U UU U S9 Ul[X!5Ul SUl g)N)rWrbrUr|rYrXr\r[r{F) r;r<rrr5r base_midblockrsr ctrl_midblockr6r) r>rxryrUrirrWrrrYr[r\r?s r0r<(ControlNetXSCrossAttnMidBlock2D.__init__s +=H4)E%') 3 8"7-  5)E%5&'-M=#@AC[!4 8"7-  "+=H&+#r/rrcURnURnURnSnURnURn[ UR SR5nURSRRn URSRRn URSRRn U"U5Rn U"U5Rn U"U5RnU"U5RnUR SR nU"UUU U U UU U UUUS9 nURR#UR%55 UR&R#UR%55 UR(R#UR%55 URR#UR%55 U$)NcNURSRSR$rr)rHs r0rOControlNetXSCrossAttnMidBlock2D.from_modules..get_first_cross_attentions$&&q)<>JJ "**1-33>> '//288CC#<]#K#Q#Q #<]#K#Q#Q 7 FZZ4]CTT - 8 8 ; Q Q'''&%4)E%=%= 3-"7   **<+B+B+DE ++M,D,D,FG ++M,D,D,FG **<+B+B+DE r/r.cUR5H nSUlM URR5H nSUlM g)r0TFN)r1r2r)r>r4s r0r32ControlNetXSCrossAttnMidBlock2D.freeze_base_paramss@__&E"&E '''224E"'E 5r/ryr{rfrzrhrlrkrroc TUb(URSS5b[RS5 Un Un UUUUUS.n U (a%[R"XR U 5/SS9n UR "U 40U D6n U (a)UR"U 40U D6n XRU 5U--n X4$)Nrr)r{rfrkrlrr!rw) rrrrrr5rrr6) r>ryr{rfrzrhrlrkrrorr joint_argss r0r'ControlNetXSCrossAttnMidBlock2D.forwards " -%))'48Dtu##%:,&<&<   YY(9(9&(ABJF##F9j9 ''=*=F//7:LLLF~r/)rr5rr6r) Nrrr!r!r!rFTr)NrNNNT)r(r)r*r+rorr<rrrFrr3rrrrrrrr.rCrDs@r0rrs} %)!(*,-/0/0*.!&-12,2,2,Tz 2,  2, #& 2,'*2,#&*2,#&*2,!4Z2,2, $d{2,2,h...3..` ( -1+.8<(,04""""" & " #TM " "DL "!%S#X 5" "!' "" vv~ ""r/rc<^\rSrSrS$S\S\S\S\\S\S\S \S-S \S \S \S \S\S\S-4U4Sjjjr\S\ S\ 4Sj5r S%Sjr S&S\ S\\ S4S\\ S4S\ S\ S-S\S-S\\\4S-S\ S-S\S-S \ S-S!\S\ 4S"jjrS#rU=r$)'ri>Nrbrcr~rrUrirrWrXrYrr[r\c>[TU]5 /n/n/nSnXlXl[ U [ 5(aU /U-n [ U5HnUUS- :XaUOUnUS:XaUOUnUR[UUU55 UR[UU-UUUS95 U(dM_UR[U X*-UU UU UU US95 M [R"U5Ul U(a[R"U5OS/U-Ul[R"U5UlU (a[!USUS9UlOSUlSUlXplg) Nrr!rrrgT)rlrcF)r;r<has_cross_attentionrXrnrorprqrrrrrrAr4r7r6r upsamplersrr)r>rbrcr~rrUrirrtrWrXrYrr[r\r4r7r6rhrures_skip_channelsrr?s r0r<'ControlNetXSCrossAttnUpBlock2D.__init__?sd"    #0 #6 2C 8 8,H+IJ+V (z"A01Z!^0C , 89Q!4L    /A!/DFX Y Z NN 25F F!-"/*  }!!&+$;$0#?#B,?.C)9(7  !#:}}W- 7D"-- 34&S]J]MM,7 (S_`DO"DO&+#,r/ base_upblock ctrl_upblockcURnSnURSRnURSRU- nURSRU- nUVs/sHoRPM n nURSRR n URSR Rn URn [US5(atSn [URSR5nU"U5RnU"U5RnU"U5RnURSR nO Sn SnSnSnSnSnUR"SLnU"UUUU U U U U UUUUUUS9nURR%URR'55 U (a3URR%URR'55 U(a6UR"R%UR"SR'55 URR%UR'55 U$s snf)NcNURSRSR$rrrs r0rNControlNetXSCrossAttnUpBlock2D.from_modules..get_first_cross_attentionrr/rrr7TF)rbrcr~rrUrirrtrWrXrYrr[r\)r6r4rcrbrrrrrr$rr7rrrYr[r\rr"r#)rrrctrl_to_base_skip_connectionsrrcrbprev_output_channelsrctrl_skip_channelssrUrrrtrWrXrYr[r\rrs r0r+ControlNetXSCrossAttnUpBlock2D.from_moduless>(4(A(A% C$++A.;; "**2.::\I +33A6BB\Q6ST6S}}6ST$,,Q/==II !))!,22== %44 < . . M+.|/F/Fq/I/\/\+] (";L"I"O"O ";L"I"]"] 8FWW $0$;$;A$>$T$T !!M+/ ("& "& # $( !#..d: #% 42'&)')E 3 3%-"7 $  %%l&:&:&E&E&GH     , ,\-D-D-O-O-Q R     , ,\-D-DQ-G-R-R-T U **+H+S+S+UV ]Us%Ir.cUR5H nSUlM UR/n[UR[ R 5(aURUR5 URbURUR5 UH!nUR5H nSUlM M# gr) r1r2r4rnr7rrArqrrs r0r31ControlNetXSCrossAttnUpBlock2D.freeze_base_paramss__&E"&E 'll^ door}} 5 5   doo . ?? &   doo .D*&+#+r/r|r}.r~r{rfrhrlrk upsample_sizerroc ^^Ub(URSS5b[RS5 [TSS5=(a5 [TSS5=(a! [TSS5=(a [TSS5mUU4Sjn [ TR TR TR[U5[U55HupnnnU (a X"U5U-- nU "UU5unn[R"UU/SS 9n[R"5(a$TR(aTRXU5nOU "X5nUcMU"UUUUU S S 9S nM TRbTRX5nU$) NrrrKrLrMrNc >T(aA[TRUUTRTRTRTR S9$X4$)N)rKrLrMrN)r rrKrLrMrN)r| res_h_baseis_freeu_enabledr>s r0maybe_apply_freeu_to_subblockMControlNetXSCrossAttnUpBlock2D.forward..maybe_apply_freeu_to_subblocksI"''!wwwwwwww%00r/r!rwFrr)rrrr!r%r4r7r6rrrrrrr)r>r|r}r~r{rfrhrlrkrrrorresnetattnrr res_h_ctrlrs` @r0r&ControlNetXSCrossAttnUpBlock2D.forwards{ " -%))'48Dtu D$ % *dD) *dD) *dD)  1:= LL OO    1 2 1 2 : 5F#z:Z3E!EE (EmU_(` %M:!II}j&AqIM$$&&4+F+F $ A A&Y] ^ &} ;  $!*?+A#1+A % !! ': 8 ?? & OOMIMr/)r7r6rrrXr4rr) rNTr!r!rTFTr)NrNNNNT)r(r)r*r+rorrr<rrrLrr3rrrrrrrr.rCrDs@r0rr>s "%),-#$#'!!&-1E-E-E-! E- !I E-  E-E-d E-'*E-!E-!E-E-E- $d{E-E-N8(:8Jd88t,.04+.8<(,$(04"GG',FCK&8G',FCK&8 G  G &} G"DLG!%S#X 5G GTzG!' GG GGr/rc @[[R"XSSS95$)Nr!r)r) zero_modulerrB)rbrcs r0rrrrs ryyAqI JJr/crUR5H"n[RRU5 M$ U$r:)r1rinitzeros_)r_ps r0r r  s*     q! Mr/cVUnX :aU$US:waX-nUS:XaU$US-nUS:waMgg)Nrr!r')numberr_factorresiduals r0rsrs&s> F  A+? q=M!  A+r/)rTr!r!rTFT)Nrr!r!rFTr:)B dataclassesrmathrtypingrrrrconfiguration_utilsr r utilsr r utils.torch_utilsr attentionrattention_processorrrrrrr embeddingsrrmodeling_utilsrunets.unet_2d_blocksrrrrrrrunets.unet_2d_conditionr r r" get_loggerr(rr$Moduler2rFrLrorrrwr}rrrrrrrrrr rsr'r/r0r$s" B(,&6';7   H %     (299(()")))))')<=&'&*$))-X!X!X!X! X!  X! t X!#&c "2T"9X!tX!tX!X!TkX! $;X!|!%&(()&'&*""& p p p: pt p #& p t pt p p  pF P P PS  PH *nkH V 1J  1DdFryydFN SbiiSl[RYY[|K r/