3j
S SK r S SKrS SKJrJr S SKrS SKrS SKrS SK J
r
J r SSK J
r
SSKJr SSKJrJr SSKJr SS KJrJrJr SS
KJr S S KJ r S
SK J r S
SK J r \" 5 ( a S SK J!s J"r# Sr$OSr$\RJ " \&5 r'Sr(S\)S\)S\*4S jr+ S"S\)S- S\,\RZ - S- S\.\) S- S\.\* S- 4S jjr/ S#S \R` S \Rb S- S \,4S jjr2 " S S!\ \5 r3g)$ N)AnyCallable)Qwen2TokenizerFastQwen3ForCausalLM )PipelineImageInput)Flux2LoraLoaderMixin)AutoencoderKLFlux2Flux2Transformer2DModel) FlowMatchEulerDiscreteScheduler)is_torch_xla_availableloggingreplace_example_docstring) randn_tensor )DiffusionPipeline )Flux2ImageProcessor)Flux2PipelineOutputTFa
Examples:
# Inpainting with text only
```py
>>> import torch
>>> from diffusers import Flux2KleinInpaintPipeline
>>> from diffusers.utils import load_image
>>> pipe = Flux2KleinInpaintPipeline.from_pretrained(
... "black-forest-labs/FLUX.2-klein-base-9B", torch_dtype=torch.bfloat16
... )
>>> pipe.to("cuda")
>>> prompt = "Face of a yellow cat, high resolution, sitting on a park bench"
>>> img_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"
>>> mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"
>>> source = load_image(img_url)
>>> mask = load_image(mask_url)
>>> image = pipe(prompt=prompt, image=source, mask_image=mask).images[0]
>>> image.save("flux2klein_inpainting.png")
```
# Inpainting with image reference conditioning
```py
>>> import torch
>>> from diffusers import Flux2KleinInpaintPipeline
>>> from diffusers.utils import load_image
>>> pipe = Flux2KleinInpaintPipeline.from_pretrained(
... "black-forest-labs/FLUX.2-klein-base-9B", torch_dtype=torch.bfloat16
... )
>>> pipe.to("cuda")
>>> prompt = "Replace this ball"
>>> img_url = "https://huggingface.co/datasets/diffusers/docs-images/resolve/main/klein_inpaint/the-ball-stadion-football-the-pitch-39362.jpeg"
>>> mask_url = "https://huggingface.co/datasets/diffusers/docs-images/resolve/main/klein_inpaint/ball_mask.png"
>>> image_reference_url = (
... "https://huggingface.co/datasets/diffusers/docs-images/resolve/main/klein_inpaint/ball.jpg"
... )
>>> source = load_image(img_url)
>>> mask = load_image(mask_url)
>>> image_reference = load_image(image_reference_url)
>>> mask = pipe.mask_processor.blur(mask, blur_factor=12)
>>> image = pipe(
... prompt=prompt, image=source, mask_image=mask, image_reference=image_reference, strength=1.0
... ).images[0]
>>> image.save("flux2klein_inpainting_ref.png")
```
image_seq_len num_stepsreturnc Su p#Su pEU S: a X@- U- n[ U5 $ X@- U- nX - U- nXx-
S- n USU - -
n
X- U
- n[ U5 $ )N)gT ?gŒ_?)g w:/&?gDw:?i g g@g i@)float) r r a1b1a2b2mum_200m_10abs p/home/wildlama/miniconda3/lib/python3.13/site-packages/diffusers/pipelines/flux2/pipeline_flux2_klein_inpaint.pycompute_empirical_mur% b sy
'FB
#FBt
"
$Ry #E
"D A
A
B 9 num_inference_stepsdevice timestepssigmasc Ub Ub [ S5 eUb S[ [ R " U R 5 R
R
5 5 ; nU( d [ SU R S35 eU R " S
X2S.UD6 U R n[ U5 nX14$ Ub S[ [ R " U R 5 R
R
5 5 ; nU( d [ SU R S35 eU R " S
XBS.UD6 U R n[ U5 nX14$ U R " U4S U0UD6 U R nX14$ ) a
Calls the scheduler's `set_timesteps` method and retrieves timesteps from the scheduler after the call. Handles
custom timesteps. Any kwargs will be supplied to `scheduler.set_timesteps`.
Args:
scheduler (`SchedulerMixin`):
The scheduler to get timesteps from.
num_inference_steps (`int`):
The number of diffusion steps used when generating samples with a pre-trained model. If used, `timesteps`
must be `None`.
device (`str` or `torch.device`, *optional*):
The device to which the timesteps should be moved to. If `None`, the timesteps are not moved.
timesteps (`list[int]`, *optional*):
Custom timesteps used to override the timestep spacing strategy of the scheduler. If `timesteps` is passed,
`num_inference_steps` and `sigmas` must be `None`.
sigmas (`list[float]`, *optional*):
Custom sigmas used to override the timestep spacing strategy of the scheduler. If `sigmas` is passed,
`num_inference_steps` and `timesteps` must be `None`.
Returns:
`tuple[torch.Tensor, int]`: A tuple where the first element is the timestep schedule from the scheduler and the
second element is the number of inference steps.
zYOnly one of `timesteps` or `sigmas` can be passed. Please choose one to set custom valuesr) z The current scheduler class zx's `set_timesteps` does not support custom timestep schedules. Please check whether you are using the correct scheduler.)r) r( r* zv's `set_timesteps` does not support custom sigmas schedules. Please check whether you are using the correct scheduler.)r* r( r( )
ValueErrorsetinspect signature
set_timesteps
parameterskeys __class__r) len) schedulerr' r( r) r* kwargsaccepts_timesteps
accept_sigmass r$ retrieve_timestepsr: u s} > !3tuu '3w/@/@AXAX/Y/d/d/i/i/k+l l .y/B/B.C Da b
M)MfM '' !)n ))
C(9(9):Q:Q(R(](](b(b(d$ee
.y/B/B.C D_ `
GvGG '' !)n )) 3MFMfM '' ))r& encoder_output generator sample_modec [ U S5 ( a! US:X a U R R U5 $ [ U S5 ( a US:X a U R R 5 $ [ U S5 ( a U R $ [ S5 e)N latent_distsampleargmaxlatentsz3Could not access latents of provided encoder_output)hasattrr? r@ moderB AttributeError)r; r< r= s r$ retrieve_latentsrF s} ~}--+2I ))00;; / /K84K ))..00 + + %%% RSSr& c 3 ^ \ rS rSrSrSrSS/r SOS\S\S\ S \
S
\ S \ 4 U 4S jjjr
\ SPS\ S \
S\\\ - S\R$ S
- S\R& S
- S\S\\ 4S jj5 r\ SQS\R, S\R, S
- 4S jj5 r\S\R, 4S j5 r\ SRS\\R, S\S \4S jj5 r\S 5 r\S 5 r\S 5 r \S\R, S!\R, S"\\R, 4S# j5 r \S$\ S"\ \\4 4S% j5 r SSS\\\ - S\R& S
- S'\S\R, S
- S\S(\ \ 4 S) jjr!S$\R, S*\RD 4S+ jr# STS*\RD \\RD - S
- S\R, S
- 4S, jjr$S-\\R, S*\RD 4S. jr%S/ r&S0 r' SUS1 jr(\)S2 5 r*\)S3 5 r+\)S4 5 r,\)S5 5 r-\)S6 5 r.\)S7 5 r/\R` " 5 \1" \25 S
S
S
S
S
S
S
S8S9S
S:S&S
S
S
S
S;S\ S
- S?\S
- S@\S
- SA\S
- SB\3SC\SD\\3 S
- SE\3S'\S*\RD \\RD - S
- S\R, S
- S\R, S
- SF\R, S
- SG\SH\ SI\4\\54 S
- SJ\6\\\4/S
4 S
- SK\\ S\S(\ \SL4 4.SM jj5 5 r7SNr8U =r9$ )VFlux2KleinInpaintPipeline a
Flux2 Klein pipeline for image inpainting with optional reference image conditioning.
Reference:
[https://bfl.ai/blog/flux2-klein-towards-interactive-visual-intelligence](https://bfl.ai/blog/flux2-klein-towards-interactive-visual-intelligence)
Args:
transformer ([`Flux2Transformer2DModel`]):
Conditional Transformer (MMDiT) architecture to denoise the encoded image latents.
scheduler ([`FlowMatchEulerDiscreteScheduler`]):
A scheduler to be used in combination with `transformer` to denoise the encoded image latents.
vae ([`AutoencoderKLFlux2`]):
Variational Auto-Encoder (VAE) Model to encode and decode images to and from latent representations.
text_encoder ([`Qwen3ForCausalLM`]):
[Qwen3ForCausalLM](https://huggingface.co/docs/transformers/en/model_doc/qwen3#transformers.Qwen3ForCausalLM)
tokenizer (`Qwen2TokenizerFast`):
Tokenizer of class
[Qwen2TokenizerFast](https://huggingface.co/docs/transformers/en/model_doc/qwen2#transformers.Qwen2TokenizerFast).
z text_encoder->transformer->vaerB
prompt_embedsr6 vae text_encoder tokenizer transformer is_distilledc $ > [ TU ] 5 U R UUUUUS9 U R US9 [ U SS 5 ( a/ S[ U R R R 5 S-
- OSU l [ U SS 5 ( a U R R R OSU l
[ U R S- U R S9U l [ U R S- U R S S
S S
S 9U l
S U l S
U l g )N)rK rL rM r6 rN )rO rK r r )vae_scale_factorvae_latent_channelsFT)rS rT do_normalize do_binarizedo_convert_rgbdo_convert_grayscale )super__init__register_modulesregister_to_configgetattrr5 rK configblock_out_channelsrS latent_channelsr image_processormask_processortokenizer_max_lengthdefault_sample_size)selfr6 rK rL rM rN rO r4 s r$ r\ "Flux2KleinInpaintPipeline.__init__ s % #
\ :V]^bdikoVpVpc$((//*L*L&MPQ&Q Rvw CJ$PUW[B\B\txx > >bd 2 ! 2 2Q 6DL`L`
2 ! 2 2Q 6 $ 4 4 !%
%(!#& r& NrY promptdtyper( max_sequence_lengthhidden_states_layersc Uc U R OUnUc U R OUn[ U[ 5 ( a U/OUn/ n/ nU HM n SU S./n
UR U
SSSS9n U" U SSSUS9n UR U S 5 UR U S
5 MO [ R " US S 9R U5 n
[ R " US S 9R U5 nU " U
USSS
9n[ R " U Vs/ s H nUR U PM snSS 9nUR X4S9nUR u nnnnUR S SSS5 R UUUU- 5 nU$ s snf )Nuser)rolecontentFT)tokenizeadd_generation_promptenable_thinkingpt
max_length)return_tensorspadding
truncationry input_idsattention_maskr dim)r} r~ output_hidden_states use_cacher )rn r( r r )rn r(
isinstancestrapply_chat_templateappendtorchcattostack
hidden_statesshapepermutereshape)rL rM rm rn r( ro rp
all_input_idsall_attention_masks
single_promptmessagestextinputsr} r~ outputkout
batch_size num_channelsseq_len
hidden_dimrJ s r$ _get_qwen3_prompt_embeds2Flux2KleinInpaintPipeline._get_qwen3_prompt_embeds s ',m ""(. $$F ' 4 4&&
#M!'MBCH 00 &* % 1 D #$ . F
!4 5 & &v.>'? @# $& IIm366v> #6A>AA&I )!%
kk -.. J4P]K^J__` ab b ! A$%GG !CF hZ(--b1A((ELL,@%,,uBUW\WcWcdeWfgEax3, *11*b"E$++E2! "7 q# A %H hZ055b9A OO ,,Q V0DellSXFY[`[g[ghi[jk ' (2 *11*b"E$++E2!/ 2 !99%9qA r& c U R u pp4U R XUS- SUS- S5 n U R SSSSSS5 n U R XS- US- US- 5 n U $ )Nr r r r )r r r r rB r num_channels_latentsr r s r$ _patchify_latents+Flux2KleinInpaintPipeline._patchify_latents su ;B--7
&,,z1aQVZ[Q[]^_//!Q1a3//*Q.FRS UZ^_U_`r& c U R u pp4U R XS- SSX45 n U R SSSSSS5 n U R XS- US- US- 5 n U $ )Nr r r r r r r r r r s r$ _unpatchify_latents-Flux2KleinInpaintPipeline._unpatchify_latents sn ;B--7
&//*u.MqRSU[c//!Q1a3//*u.MvXYz[`cd[der& c n U R u pp4U R XX4- 5 R SSS5 n U $ )zg
pack latents: (batch_size, num_channels, height, width) -> (batch_size, height * width, num_channels)
r r r r )rB r r r r s r$
_pack_latents'Flux2KleinInpaintPipeline._pack_latents s; 3:--/
&//*FNKSSTUWXZ[\r& r r c / n[ X5 GH- u p4UR u pVUSS2S4 R [ R 5 nUSS2S4 R [ R 5 n[ R
" U5 S- n [ R
" U5 S- n
Xz- U- n [ R " X- U4UR UR S9n U R SU R S5 R SU5 U5 U R XU5 R SSS5 n UR U 5 GM0 [ R " USS9$ )z1
using position ids to scatter tokens into place
Nr r r( rn r r r )zipr r r int64maxzerosr( rn scatter_r r r r r r )
r r x_listdataposr chh_idsw_idsr r flat_idsr s
r$ _unpack_latents_with_ids2Flux2KleinInpaintPipeline._unpack_latents_with_ids s Q IDJJEA1ILL-E1ILL-E % 1$A % 1$A y5(H++qubk$++TZZPC LLH . .q 1 8 8R @$ G ((1$,,Q15C MM# ! '$ {{6q))r& imagec [ U [ 5 ( a U S n [ U [ R R 5 ( a U R U R
4$ [ U [ R 5 ( a U R S U R S 4$ [ U [ R 5 ( aL U R S: a U R S U R S 4$ U R S U R S 4$ [ U S5 ( a U R S U R S 4$ [ S[ U 5 35 e)z7Helper to get (height, width) without rounding/scaling.r r r r zUnsupported image type: )r r PILImager r r Tensorr npndarrayndimrC r- r )r s r$ _get_raw_image_size-Flux2KleinInpaintPipeline._get_raw_image_size s eT " "!HE eSYY__ - -<<, ,
u||
,
,;;r?EKKO3 3
rzz
*
*zzQ {{2 B77;;r?EKKO3 3 5' " ";;r?EKKO3 33DK=ABBr& r num_images_per_prompttext_encoder_out_layersc x U=( d U R nUc Sn[ U[ 5 ( a U/OUnUc( U R U R U R
UUUUS9nUR u pxn UR SUS5 nUR Xs- US5 nU R U5 n
U
R U5 n
XJ4$ )N )rL rM rm r( ro rp r r ) _execution_devicer r r rL rM r repeatr r r ) rg rm r( r rJ ro r r r r text_idss r$
encode_prompt'Flux2KleinInpaintPipeline.encode_prompt s 1411 >F ' 4 4&& 9 9 ! . . .. $7%<
: M "/!4!4
Q%,,Q0EqI
%**:+MwXZ[
))-8;;v& &&r& r< c UR S:w a [ SUR S35 e[ U R R U5 USS9nU R U5 nU R R R R SSSS5 R UR UR 5 n[ R " U R R R R SSSS5 U R R R - 5 R UR UR 5 nX4-
U- nU$ )Nr zExpected image dims 4, got r rA )r< r= r r )r r- rF rK encoder bn running_meanr r r( rn r sqrt running_varr` batch_norm_eps)rg r r< r latents_bn_meanlatents_bn_stds r$ _encode_vae_image+Flux2KleinInpaintPipeline._encode_vae_image s ::? :5::,a HI I( )?9bjk
..}=
((++22772q!DGG
H\H\^k^q^qr DHHKK$;$;$@$@B1$MPTPXPXP_P_PnPn$norr -"5"5
'8NJ
r& c [ U [ 5 ( a* [ U 5 U:w a [ S[ U 5 SU S35 eS[ U5 U R
S- - - nS[ U5 U R
S- - - nX4S- US- US- 4n [ R " XUS9n U R U 5 n
U
R U5 n
UR XS9nUR S U R S- :w a U R XS9nOUnU R R R R! SS SS5 R UR" UR$ 5 n[ R& " U R R R( R! SS SS5 U R R* R, - 5 R UR" UR$ 5 nX-
U- nX>R S
: a@ X>R S
- S
:X a+ X>R S
- n[ R. " U/U- S
S 9nOFX>R S
: a4 X>R S
- S
:w a [ S UR S
S
U S35 eU
c' [1 XXS9nU R2 R5 XU5 n
OU
R U5 nUn
U R7 U5 nU R7 U5 nU R7 U
5 n
U
UUX4$ )Nz/You have passed a list of generators of length z+, but requested an effective batch size of z@. Make sure the batch size matches the length of the generators.r r r r r r< r r r z'Cannot duplicate `image` of batch size z to z text prompts.)r< r( rn ) r r r5 r- intrS r r r r r rb r rK r r r r( rn r r r` r r r r6 scale_noiser )rg r timestepr num_latents_channelsr r rn r( r< rB r
dummy_latentslatent_image_idsr r r additional_image_per_promptnoisepacked_image_latentss r$ prepare_latents)Flux2KleinInpaintPipeline.prepare_latents- s i & &3y>Z+G A#i.AQ R &<'gi
c&kd&;&;a&?@ASZD$9$9A$=>?A 5v{EQJO EF
33MB+..v64 ;;q>T 1 1A 5 5 2 2 2 TM !M "hhkk 6 6 ; ;Ar1a H K KML`L`bobubu vO "ZZ (?(?(D(DQAq(QTXT\T\TcTcTrTr(r s v v $$m&9&9 N + < NM ++A. .:@S@STU@V3VZ[3[*48K8KA8N*N ' !II}o8S&SYZ [M
--a0
0ZBUBUVWBX5X\]5] 9-:M:Ma:P9QQUV`Uaaop
? FXEnn00 %PGJJv&EG""5) # 1 1- @ $$W- 3]TTr& imagesc n / nU HU nUR XES9nUR S U R S- :w a U R XsS9nOUnUR U5 MW U R Xb5 n / n
U H n U R
U 5 n U R S n
U
S:X a US: a U R USS5 n OCX:X a O=U R S nU R SX- S5 n US: a U R USS5 n U
R U 5 M [ R " U
SS9nU R U5 n Xi4$ )Nr r r r r r r ) r r rb r r r r r r r r )rg r r r< r( rn r r image_latentr
final_latentslatentpackedr r s r$ prepare_image_latents/Flux2KleinInpaintPipeline.prepare_image_latentsk sC
EHHFH8E{{1~!5!5!99 # 5 5E 5 W $ . 22=M
#F''/F,,q/CaxJN z1a8" !,,q/ 3="= > #]]:q! >#4 5 5 NN * *7^^5I5I+I J777r& c
^ Uc [ S5 eUc [ S5 eUS: d US: a [ SU 35 eUb UT R S- - S:w d UbB UT R S- - S:w a, [ R ST R S- SU S U S
35 U
bW [ U 4S jU
5 5 ( d= [ S T R
S
U
V
s/ s H oT R
; d M U
PM sn
35 eUb U b [ SU SU S35 eUc U c [ S5 eUbA [
U[ 5 ( d, [
U[ 5 ( d [ S[ U5 35 eU b [
U[ R R 5 ( d [ S[ U5 S35 e[
U[ R R 5 ( d [ S[ U5 S35 eUS:w a [ SU S35 eOUbe [
U[ R R [ R [ R [ 45 ( d [ S[ U5 S35 eUbe [
U[ R R [ R [ R [ 45 ( d [ S[ U5 S35 eUbe [
U[ R R [ R [ R [ 45 ( d [ S[ U5 S35 eU S: a6 T R R" ( a [ R S U S 35 g g g s sn
f ) Nz*`image` has to be provided for inpainting.z/`mask_image` has to be provided for inpainting.r r z2The value of strength should in [0.0, 1.0] but is r z-`height` and `width` have to be divisible by z but are z and z(. Dimensions will be resized accordinglyc 3 @ ># U H oTR ; v M g 7fN)_callback_tensor_inputs).0r rg s r$ 9Flux2KleinInpaintPipeline.check_inputs.. s F
7Y!-- -7Ys z2`callback_on_step_end_tensor_inputs` has to be in z , but found z Cannot forward both `prompt`: z and `prompt_embeds`: z2. Please make sure to only forward one of the two.zeProvide either `prompt` or `prompt_embeds`. Cannot leave both `prompt` and `prompt_embeds` undefined.z2`prompt` has to be of type `str` or `list` but is zJThe image should be a PIL image when inpainting mask crop, but is of type r zOThe mask image should be a PIL image when inpainting mask crop, but is of type pilz@The output type should be PIL when inpainting mask crop, but is z\`image` has to be of type `PIL.Image.Image`, `torch.Tensor`, `np.ndarray`, or `list` but is za`mask_image` has to be of type `PIL.Image.Image`, `torch.Tensor`, `np.ndarray`, or `list` but is zf`image_reference` has to be of type `PIL.Image.Image`, `torch.Tensor`, `np.ndarray`, or `list` but is ?zGuidance scale z+ is ignored for step-wise distilled models.)r- rS loggerwarningallr4 r r r r r r r r r r r` rO )rg rm r
mask_imageimage_referencer, r r output_typerJ "callback_on_step_end_tensor_inputspadding_mask_cropguidance_scaler s ` r$ check_inputs&Flux2KleinInpaintPipeline.check_inputs s = IJ J NO O a<8a< QRZQ[ \] ]
$//!349 ..23q8 NN?@U@UXY@Y?ZZcdjckkpqvpw x` a
. 9# F
7YF
C
C
DTEaEaDbbn |^ pH |^vw ko kG kG bGpq |^ pH oI J
-"; 08N}o ^0 0
^
5 w
FC)@)@TZ\`IaIa QRVW]R^Q_ `a a ( eSYY__55 `aefkal`mmno j#))//:: Z(), e# #cdocppq!rss $ !%#))//5<<UY)Z[[$##';-q2 %!*syy bjjZ^._``$##'
#3"4A7
& o rzz[_/`aa #O 4 5Q8
C DKK$<$< NN_^,< r= r r rA r, r' r* rB negative_prompt_embedsr? return_dictrN callback_on_step_endr@ .c " U=( d U R U R - nU=( d U R U R - nU R UUUUUUUUUUUU S9 Xl UU l SU l SU l U R S- n[ U[ R 5 ( az UR S:X aj UR S5 U R S- :X aH UnUnSnSn UR S U R - S- nUR S U R - S- nGOtUGbp [ U[ 5 ( aJ [ US [ R 5 ( a( US R S:X a [ R " US S
9n[ U[ 5 ( a US OUn U R! U 5 u n n U U - S : a [" R$ " S U U - - 5 n U R& R) U[+ U U - 5 [+ U U - 5 5 n[ U[ 5 ( a US OUn U R! U 5 u n n U U- U- n!U U- U- n"U!nU"nUb U R, R/ XFXWS 9nS
n OSnSn UnU R& R1 UU"U!UU S9nSn#UGb [ U[ R 5 ( a# UR S5 U R S- :X GdF [ U[ 5 ( aJ [ US [ R 5 ( a( US R S:X a [ R " US S
9n[ U[ 5 ( a US OUn$U R! U$5 u n%n&U%U&- S : a [" R$ " S U%U&- - 5 n U R& R) U[+ U%U - 5 [+ U&U - 5 5 n[ U[ 5 ( a US OUn$U R! U$5 u n%n&U&U- U- n'U%U- U- n(U R& R1 UU(U'SS9n#OUb U R2 R4 R6 R9 SSSS5 R; UR< UR> 5 n)[ R$ " U R2 R4 R@ R9 SSSS5 U R2 RB RD - 5 R; UR< UR> 5 n*UU)-
U*- n#Ub [ U[F 5 ( a Sn+O3Ub! [ U[ 5 ( a [I U5 n+OUR S n+U RJ n,U RM UUU,U UUS9u nn-U RN ( a@ Sn.Ub$ [ U[ 5 ( a U./[I U5 - n.U RM U.UU,U UUS9u nn/U
c [P RR " SSU - U 5 OU
n
[U U RV RB S5 ( a' U RV RB RX ( a Sn
[+ U5 U R - S- [+ U5 U R - S- - n0[[ U0U S9n1[] U RV U U,U
U1S9u n2n U R_ XU,5 u n2n U S: a [a SU SU S35 eU2SS Rc U+U - 5 n3U Rd RB Rf S- n4U Ri WU3U+U - U4UUUR> U,U
U5
u nn5n6n7n8U7/n9U#b U9Rk U#5 U Rm U9U+U - U
U,UR> 5 u n:n;U R, R1 XEUW WS9n U,5 n=[q [I U25 XRV Rr - -
S 5 n>[I U25 U l: [ R " U8U;/SS
9n?U Rw U S 9 n@[y U25 GH u nAnBU Rz ( a M WBU l UBR} UR S 5 R; UR> 5 nC[ R " UU:/SS
9nDU?nEUDR; U Rd R> 5 nDU Rd R S 5 U Re UDUCS - SUU-UEU R SS 9S nFSSS5 WFSS2SUR S5 24 nFU RN ( ao U Rd R S 5 U Re WDWCS - SUW/WEU R SS 9S nGSSS5 WGSS2SUR S5 24 nGUGU WFUG-
- - nFUR> nHU RV R WFWBUSS!9S nU6nIU=nJWA[I U25 S-
: a: U2WAS- nKU RV R WI[ R " UK/5 U55 nISWJ-
WI- UJU- - nUR> WH:w a> [ R R R 5 ( a UR; WH5 nUbJ 0 nLU H nM[ 5 UM WLUM' M U" U WAWBWL5 nNUNR S"U5 nUNR S#U5 nWA[I U25 S-
:X d) WAS- U>: a0 WAS- U RV Rr - S :X a W@R 5 [ ( d GM [ R " 5 GM SSS5 SU l U R UU85 nU R2 R4 R6 R9 SSSS5 R; UR< UR> 5 nO[ R$ " U R2 R4 R@ R9 SSSS5 U R2 RB RD - 5 R; UR< UR> 5 nPUUP- UO- nU R U5 nUS$:X a UnOhU R2 R USS!9S nU R& R UUS%9nUb. U VAs/ s H! nAU R& R UWUAU5 PM# nnAU R 5 U( d U4$ [ US&9$ ! , ( d f GN= f! , ( d f GN= f! , ( d f GN= fs snAf )'u8!
Function invoked when calling the pipeline for inpainting.
Args:
prompt (`str` or `List[str]`, *optional*):
The prompt or prompts to guide the image generation. If not defined, one has to pass `prompt_embeds`.
instead.
image (`torch.Tensor`, `PIL.Image.Image`, `np.ndarray`, `List[torch.Tensor]`, `List[PIL.Image.Image]`, or `List[np.ndarray]`):
`Image`, numpy array or tensor representing an image batch to be used as the starting point. For both
numpy array and pytorch tensor, the expected value range is between `[0, 1]`. If it's a tensor or a
list of tensors, the expected shape should be `(B, C, H, W)` or `(C, H, W)`. If it is a numpy array or
a list of arrays, the expected shape should be `(B, H, W, C)` or `(H, W, C)`. It can also accept image
latents directly, in which case encoding is skipped. Latents must be in patchified form of shape `(B,
latent_channels * 4, H // 2, W // 2)`, where each 2×2 spatial patch has been folded into the channel
dimension.
image_reference (`torch.Tensor`, `PIL.Image.Image`, `np.ndarray`, `List[torch.Tensor]`, `List[PIL.Image.Image]`, or `List[np.ndarray]`, *optional*):
`Image`, numpy array or tensor representing an image batch to be used as the reference for the masked
area. This allows conditioning the inpainted region on a specific reference image. For both numpy array
and pytorch tensor, the expected value range is between `[0, 1]`. If it's a tensor or a list of
tensors, the expected shape should be `(B, C, H, W)` or `(C, H, W)`. If it is a numpy array or a list
of arrays, the expected shape should be `(B, H, W, C)` or `(H, W, C)`. It can also accept image latents
directly, in which case encoding is skipped. Latents must be in patchified form of shape `(B,
latent_channels * 4, H // 2, W // 2)`, where each 2×2 spatial patch has been folded into the channel
dimension.
mask_image (`torch.Tensor`, `PIL.Image.Image`, `np.ndarray`, `List[torch.Tensor]`, `List[PIL.Image.Image]`, or `List[np.ndarray]`):
`Image`, numpy array or tensor representing an image batch to mask `image`. White pixels in the mask
are repainted while black pixels are preserved. If `mask_image` is a PIL image, it is converted to a
single channel (luminance) before use. If it's a numpy array or pytorch tensor, it should contain one
color channel (L) instead of 3, so the expected shape for pytorch tensor would be `(B, 1, H, W)`, `(B,
H, W)`, `(1, H, W)`, `(H, W)`. And for numpy array would be for `(B, H, W, 1)`, `(B, H, W)`, `(H, W,
1)`, or `(H, W)`.
height (`int`, *optional*, defaults to self.default_sample_size * self.vae_scale_factor):
The height in pixels of the generated image. This is set to 1024 by default for the best results.
width (`int`, *optional*, defaults to self.default_sample_size * self.vae_scale_factor):
The width in pixels of the generated image. This is set to 1024 by default for the best results.
padding_mask_crop (`int`, *optional*, defaults to `None`):
The size of margin in the crop to be applied to the image and masking. If `None`, no crop is applied to
image and mask_image. If `padding_mask_crop` is not `None`, it will first find a rectangular region
with the same aspect ration of the image and contains all masked area, and then expand that area based
on `padding_mask_crop`. The image and mask_image will then be cropped based on the expanded area before
resizing to the original image size for inpainting. This is useful when the masked area is small while
the image is large and contain information irrelevant for inpainting, such as background.
strength (`float`, *optional*, defaults to 0.8):
Indicates extent to transform the reference `image`. Must be between 0 and 1. `image` is used as a
starting point and more noise is added the higher the `strength`. The number of denoising steps depends
on the amount of noise initially added. When `strength` is 1, added noise is maximum and the denoising
process runs for the full number of iterations specified in `num_inference_steps`. A value of 1
essentially ignores `image`.
num_inference_steps (`int`, *optional*, defaults to 50):
The number of denoising steps. More denoising steps usually lead to a higher quality image at the
expense of slower inference.
sigmas (`List[float]`, *optional*):
Custom sigmas to use for the denoising process with schedulers which support a `sigmas` argument in
their `set_timesteps` method. If not defined, the default behavior when `num_inference_steps` is passed
will be used.
guidance_scale (`float`, *optional*, defaults to 8.0):
Guidance scale as defined in [Classifier-Free Diffusion
Guidance](https://huggingface.co/papers/2207.12598). `guidance_scale` is defined as `w` of equation 2.
of [Imagen Paper](https://huggingface.co/papers/2205.11487). Guidance scale is enabled by setting
`guidance_scale > 1`. Higher guidance scale encourages to generate images that are closely linked to
the text `prompt`, usually at the expense of lower image quality. For step-wise distilled models,
`guidance_scale` is ignored.
num_images_per_prompt (`int`, *optional*, defaults to 1):
The number of images to generate per prompt.
generator (`torch.Generator` or `List[torch.Generator]`, *optional*):
One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html)
to make generation deterministic.
latents (`torch.Tensor`, *optional*):
Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for image
generation. Can be used to tweak the same generation with different prompts. If not provided, a latents
tensor will be generated by sampling using the supplied random `generator`.
prompt_embeds (`torch.Tensor`, *optional*):
Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not
provided, text embeddings will be generated from `prompt` input argument.
negative_prompt_embeds (`torch.Tensor`, *optional*):
Pre-generated negative text embeddings. Note that "" is used as the negative prompt in this pipeline.
If not provided, will be generated from "".
output_type (`str`, *optional*, defaults to `"pil"`):
The output format of the generate image. Choose between
[PIL](https://pillow.readthedocs.io/en/stable/): `PIL.Image.Image` or `np.array`.
return_dict (`bool`, *optional*, defaults to `True`):
Whether or not to return a [`~pipelines.flux2.Flux2PipelineOutput`] instead of a plain tuple.
attention_kwargs (`dict`, *optional*):
A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under
`self.processor` in
[diffusers.models.attention_processor](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py).
callback_on_step_end (`Callable`, *optional*):
A function that calls at the end of each denoising steps during the inference. The function is called
with the following arguments: `callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int,
callback_kwargs: Dict)`. `callback_kwargs` will include a list of all tensors as specified by
`callback_on_step_end_tensor_inputs`.
callback_on_step_end_tensor_inputs (`List`, *optional*):
The list of tensor inputs for the `callback_on_step_end` function. The tensors specified in the list
will be passed as `callback_kwargs` argument. You will only be able to include variables listed in the
`._callback_tensor_inputs` attribute of your pipeline class.
max_sequence_length (`int` defaults to 512): Maximum sequence length to use with the `prompt`.
text_encoder_out_layers (`Tuple[int]`):
Layer indices to use in the `text_encoder` to derive the final prompt embeddings.
Examples:
Returns:
[`~pipelines.flux2.Flux2PipelineOutput`] or `tuple`: [`~pipelines.flux2.Flux2PipelineOutput`] if
`return_dict` is True, otherwise a `tuple`. When returning a tuple, the first element is a list with the
generated images.
) rm r r= r> r, r r r? rJ r@ rA rB NFr r r defaultr r r i )padfill) crops_coords resize_modecrop)rf r )rm rJ r( r ro r r r9 use_flow_sigmas)r r )r* r z?After adjusting the num_inference_steps by strength parameter: z", the number of pipeline steps is z4 which is < 1 and not appropriate for this pipeline.)r r rf re )totalcondi )r r guidanceencoder_hidden_statestxt_idsimg_idsjoint_attention_kwargsr_ uncond)r_ rB rJ r )r? )r )Trf rS rC rF rM rV rZ r r r r r rb r r r r mathr rc resizer
rd get_crop_region
preprocessrK r r r r r( rn r r` r r r5 r r rJ r linspacerC r6 rh r% r: r/ r- r rN in_channelsr r r r& r r+ rR progress_bar enumerater[ r
cache_contextrN stepr r backendsmps is_availablelocalspopupdate
XLA_AVAILABLExm mark_stepr r decode postprocess
apply_overlaymaybe_free_model_hooksr )Qrg rm r r> r= r r rA r, r' r* rB r r< rB rJ r^ r? r_ rN r` r@ ro r multiple_of
init_imageoriginal_imagere rf imgraw_hraw_wr image_width image_heightprocessed_image_reference
img_reference raw_ref_h raw_ref_wimage_reference_widthimage_reference_heightbn_meanbn_stdr r( r negative_promptnegative_text_idsr r r) latent_timestepr r r image_latents_encodedr
ref_imagescondition_image_latentscondition_image_idsmask_conditionr# num_warmup_stepscombined_image_idsrw r r r latent_model_inputrn
noise_predneg_noise_pred
latents_dtypeinit_latents_proper init_masknoise_timestepcallback_kwargsr callback_outputsr r sQ r$ __call__"Flux2KleinInpaintPipeline.__call__5 s
N K433d6K6KKI11D4I4II
! + #'/Q /)
. !1 !% ++a/ eU\\ * *uzzQ5::a=TXThThklTlCl J "N L#K[[^d&;&;;a?FKKNT%:%::Q>E
%&&:eAh +M+MRWXYRZR_R_cdRd %Q/ ( 5 5%(5C 33C8LE5u}{* + "?@ ,,33E3uu};MsSX[`S`Oab",UD"9"9eAhu # 7 7 < u K/;>K ![ 0K?L E!F !, # 2 2 B B:V\ B t $ # ' "N --88|[|Yd 9 J
%)! & 5 5/:N:Nq:QUYUiUilmUm:m ?D11 q 15<<@@#A&++q0"'))O"C2<_d2S2SOA .YhM#'#;#;M#J Iy9${2 +Y1F"GH"&"6"6"="=#SU):%;SUAR=S# 7A RV6W6W 2]l
'+'?'?
'N$ 9%.+%=$L !&/;&>+%M "(,(<(<(G(G &% " )H ) % * ((++22772q!DGGH^H^`o`u`uv DHHKK$;$;$@$@B1$MPTPXPXP_P_PnPn$norr#**O,A,A .=w-F&,P) *VS"9"9J
Jvt$<$< VJ&,,Q/J'' #'"4"4'"7 3$;
#5 #
x + + O!j&>&>#2"3c&k"A8<8J8J& 4 &;$7(?
9K 9 5 "$5 TZSaS!&9"9;NOgm 4>>((*; < < AVAVAfAfF V(=(==Bs5zUYUjUjGjnoGop
!
I\
]); NN *
& & *.););N ,,
$ 0 7 87;7Q7Q . . 8
4 !4 ,,77 U Zf 8
(( !
s9~0CnnFZFZ0Z Z\]^ !)n #YY(8:M'NTU V
%8
9\!),1>> )*& 88GMM!$4588 G%*YY9P/QWX%Y",%7%:%:4;K;K;Q;Q%R"%%33F;!%!1!1&8!)D !%.; ( '/3/D/D$) "2 " "J < (+:J-KU+' y=,??)gBUU==M1 ~~))6688")**]";'3&(O ?-3Xa[* @';D!Q'X$ . 2 29g FG$4$8$8 -$XM I**A9I/IqSTuX\XfXfXlXlNlpqNq '') =LLNW - :\ "& //9IJ ((++22772q!DGGX_XeXef DHHKK$;$;$@$@B1$MPTPXPXP_P_PnPn$norr NNGMM
N*_<**73 ( "EHHOOGO?BE((44U 4TE ,mrmrhiD((66z>STVbcmr
##%8O"%00{ <; BA9 :
9BsK
B=s:'s.As:<'s(#F%s:
s: (t
s% s:(
s7 2s::
t ) rM rV rF rZ rR rf rc rb rd re rS )F)NNrY ri r3 )
)Nr NrY ri )NNNNNN):__name__
__module__ __qualname____firstlineno____doc__model_cpu_offload_seqr4 r r
r r r boolr\ staticmethodr r r rn r( r
r r r r r r r r r r tupler r Generatorr r r r& r/ rC propertyrB rJ rN rS rW r[ no_gradr EXAMPLE_DOC_STRINGr dictr r r __static_attributes__
__classcell__)r4 s @r$ rH rH s ( = (/ : #%'2%' %' ' %'
& %' -
%' %' %'N %)&*#&*56 &6 %6 d3i6 {{T! 6
t# 6 !
6 #3i6 6 p (,$ <<$ $$ $$ > > ELL)> > > > @ *ELL * *$u||J\ * *2 C#5 C%S/ C C. '+%&-1#&.9 'd3i ' t# ' # '
||d* ' !
' "'s 'F
u||
2 EI'+<U ??T%//%::TA<U $<U|*/U\\"*/ ?? */X D 8* +/ Wjr $ $ I I & & # # & & ]]_ 12 *.+/5904 ! (, #%%) #%&DH'+-16: 26HL9B #&3>1A1d3i$&A1 "D(A1 ,d2 A1
'- A1 d
A1 TzA1 :A1 A1 !A1 U d"A1 A1 #A1 ??T%//%::TA A1 $ A1 ||d*!A1" !& t 3#A1$ %A1& 'A1( sCx.4/)A1* 'S$'7'= > E+A1, -1I-A1. !/A10 "'sCx 1A1 3 A1r& rH r )Nr@ )4r/ rq typingr r numpyr r r transformersr r rc r loadersr modelsr
r
schedulersr utilsr
r r utils.torch_utilsr pipeline_utilsr r pipeline_outputr torch_xla.core.xla_modelcore xla_modelr r
get_loggerr r: r r
r r% r r( r r: r r rF rH r, r& r$ r s/
= 1 + A 9 O O - . 0 0 ))MM
H %1 j * '+(,"&!% 8* t8*
%,, %8* Cy4 8*
K$ 8*z `h
TLL
T-2__t-C
TY\
Tz1 13G z1r&