3jTSSKrSSKrSSKJr SSKrSSKJr SSK J r \ R"\ 5r SS\4Sjjr\R R"R$SS4S\S\4S jjr"S S \R(5r"S S \R(5r"SS\R(5r"SS\R(5r"SS\R(5rg)N)loggingkey_chunk_sizec^^^^^^^^^TRSSunmmTRSm[TU5mT[R"T5- m[R "[ RSS9U4Sj5mUUUUUUUU4Sjn[ RRU[R"SUT5S 9upxn [R"U SS S 9n [R"X- 5n U[R"U SS 9-nX-nURSS 9n [R"US5RSS 9n X- $) zBMulti-head dot product attention with a limited number of queries.NF) prevent_csecF>[R"SXTS9n[R"USSS9n[RR U5n[R "X4- 5n[R"SX%TS9n[R"SU5nXeRSS9U4$) Nz...qhd,...khd->...qhk) precisionr Taxiskeepdimsz...vhf,...qhv->...qhfz ...qhk->...qhr)jnpeinsummaxjaxlax stop_gradientexpsum)querykeyvalue attn_weights max_score exp_weights exp_valuesr s Y/home/wildlama/miniconda3/lib/python3.13/site-packages/diffusers/models/attention_flax.pysummarize_chunk/_query_chunk_attention..summarize_chunk#szz"95QZ[ GGLrDA GG)))4 ggl67 ZZ 7W`a JJ : OOO4i@@c ^>[RRTS/TRS- -USS/-[ TR SS5TTT/-S9n[RRT S/T RS- -USS/-[ T R SS5TTT /-S9nT"TX5$)Nrroperand start_indices slice_sizes)rr dynamic_slicendimlistshape) chunk_idx key_chunk value_chunk k_featuresrr num_headsrr! v_featuresrs r chunk_scanner-_query_chunk_attention..chunk_scanner0sGG))#A.)Q1BBSYYs^, :/VV* gg++#a0Iq!3DDU[["-..)Z1XX, ui==r#r)fxsTr r)r-minrsqrt functoolspartialr checkpointrmaparangerr expand_dimsr)rrrr rnum_kvr4 chunk_values chunk_weights chunk_max global_max max_diffs all_values all_weightsr1r2r!r3s````` @@@@r _query_chunk_attentionrHs$'IIbcN!FIzRJ0N CHHZ( (Es~~59 A: A>>".1WW[[=SZZXY[acqMr[-s*LT:J ./ICOOIB77LM!!q!)J//-488a8@K  ##r#iquery_chunk_sizec ^^^^^^^ ^ ^ TRSSum m m UUU U UU UUU4 Sjn[RRUSS[R "T T- 5S9upx[ R"USS9$)a Flax Memory-efficient multi-head dot product attention. https://huggingface.co/papers/2112.05682v2 https://github.com/AminRezaei0x443/memory-efficient-attention Args: query (`jnp.ndarray`): (batch..., query_length, head, query_key_depth_per_head) key (`jnp.ndarray`): (batch..., key_value_length, head, query_key_depth_per_head) value (`jnp.ndarray`): (batch..., key_value_length, head, value_depth_per_head) precision (`jax.lax.Precision`, *optional*, defaults to `jax.lax.Precision.HIGHEST`): numerical precision for computation query_chunk_size (`int`, *optional*, defaults to 1024): chunk size to divide query array value must divide query_length equally without remainder key_chunk_size (`int`, *optional*, defaults to 4096): chunk size to divide key and value array value must divide key_value_length equally without remainder Returns: (`jnp.ndarray`) with shape of (batch..., query_length, head, value_depth_per_head) rNc > [RRT S/T RS- -USS/-[ T R SS5[ T T5TT/-S9nUT -[UTT TTS94$)Nrr%rr&)rrrr r)rrr*r+r,r-r8rH) r._ query_chunkrrr2num_qr q_featuresrrIrs r r45jax_memory_efficient_attention..chunk_scannerfsgg++3%**q.1iA5FFU[["-.#6F2NPY[e1ff,  ( ( "!s%9]k   r#r)r6initr7lengthr)r-rrscanmathceilr concatenate) rrrr rIrr4rLresr2rNrOs `````` @@@r jax_memory_efficient_attentionrXOsp*$);;rs#3 E9j   WW\\   yy!112 FA ??3R ((r#c\rSrSr%Sr\\S'Sr\\S'Sr\\S'Sr \ \S 'S r \ \S 'S r \ \S '\Rr\R \S 'SrSrSrSSjrSrg) FlaxAttentiona A Flax multi-head attention module as described in: https://huggingface.co/papers/1706.03762 Parameters: query_dim (:obj:`int`): Input hidden states dimension heads (:obj:`int`, *optional*, defaults to 8): Number of heads dim_head (:obj:`int`, *optional*, defaults to 64): Hidden states dimension inside each head dropout (:obj:`float`, *optional*, defaults to 0.0): Dropout rate use_memory_efficient_attention (`bool`, *optional*, defaults to `False`): enable memory efficient attention https://huggingface.co/papers/2112.05682 split_head_dim (`bool`, *optional*, defaults to `False`): Whether to split the head dimension into a new axis for the self-attention computation. In most cases, enabling this flag should speed up the computation for Stable Diffusion 2.x and Stable Diffusion XL. dtype (:obj:`jnp.dtype`, *optional*, defaults to jnp.float32): Parameters `dtype` query_dimheads@dim_headdropoutFuse_memory_efficient_attentionsplit_head_dimdtypec[RS5 URUR-nURS-Ul[ R "USURSS9Ul[ R "USURSS9Ul [ R "USURSS9Ul [ R "URURSS 9Ul [ R"URS 9Ulg) NFlax classes are deprecated and will be removed in Diffusers v1.0.0. We recommend migrating to PyTorch classes or pinning your version of Diffusers.gFto_q)use_biasrenameto_kto_vto_out_0)rerjrate)loggerwarningr`r^scalennDensererrrr\ proj_attnDropoutrb dropout_layerself inner_dims r setupFlaxAttention.setups [ MMDJJ. ]]D( XXi%tzzPVW 88ITZZfUXXi%tzzPVW $.. TZZT\\:r#cURup#nURnURX#XTU-5n[R"US5nURX%-X4U-5nU$N)rrr%r-r^reshaper transposerytensor batch_sizeseq_lendim head_sizes r reshape_heads_to_batch_dim(FlaxAttention.reshape_heads_to_batch_dims[#)<< SJJ  Yy@PQv|4 6 AQR r#cURup#nURnURX%-XSU5n[R"US5nURX%-X4U-5nU$r~rrs r reshape_batch_dim_to_heads(FlaxAttention.reshape_batch_dim_to_headssZ#)<< SJJ  7SQv|4 7 /R r#Nc6UcUOUnURU5nURU5nURU5nUR(aURSn[ R "XGSURUR45n[ R "XWSURUR45n [ R "XgSURUR45n O3URU5nURU5n URU5n UR(aURSSS5nU RSSS5n U RSSS5n URSn U S-S:Xa[U S- 5n O;U S-S:Xa[U S- 5n O#U S-S:Xa[U S- 5n O [U 5n [XXS S 9nURSSS5nURU5nGOUR(a[ R"S X5n O[ R"S X5n XR -n ["R$"XR(aSOSS 9nUR(aW[ R"SX5nURSn[ R "XSURUR-45nO([ R"SX5nURU5nUR'U5nUR)XS9$)Nrr rrrr_i@)rIrzb t n h, b f n h -> b n f tzb i d, b j d->b i jrzb n f t, b t n h -> b f n hzb i j, b j d -> b i d deterministic)rrrrdr-rrr^r`rrcrintrXrrrrrssoftmaxrurw)ry hidden_statescontextr query_projkey_proj value_projb query_states key_states value_statesflatten_latent_dimrIattention_scoresattention_probss r __call__FlaxAttention.__call__s#*?-ZZ . 88G$ZZ(   ##A&A;;zr4::t}}3UVLX2tzz4==/QRJ;;zr4::t}}3UVL:::FL88BJ:::FL  . .'11!Q:L#--aA6J'11!Q:L ".!3!3B!7 !B&!+#&'9B'>#? #b(A-#&'9B'>#? #a'1,#&'9A'=#> #&'9#: :,jrM*33Aq!