3j<2%SSKJr SSKrSSKrSSKrSSKrSSKJr SSKJ r SSK J r J r J r SSKrSSKJr SSKJs Jr \R R-5(a SSKJs Jr SSKJrJrJrJrJrJ r J!r!J"r"J#r#J$r$J%r%J&r&J'r'J(r(J)r)J*r* SSK+J,r,J-r- SS K.J/r/J0r0 S S K1J2r2 \ (aS S K1J3r3 S r4Sr5Sr6Sr7Sr8Sr9\"\:5r;\"5=(a \ "S\45r<\"5r=\"5=(a \"S\55r>\#"5=(a \$"S\65r?\&"S\75r@\%"5rA\'"5=(a \("S\85rB\)"5=(a \*"S\95rC\<(aSSKDJErEJFrF SSKGJHrHJIrI OSrESrFSrHSrI\=(aSSKOJErP SSKOJFrQ OSrPSrQ\>(aSSKRJErS OSrS\?(aSSKTJUrUJVrVJWrWJXrXJYrYJZrZ O SrUSrXSrYSrVSrWSrZ\@(aSSK[Js J\s J]r] OSr]\A(aSS!K^J_r_ OSr_\B(aSS#K`Jarb OSrb\C(aSSKcJdre OSre\RS&:a-\RRri\RRrkOSSSS'.S(jjrlSSS S).S*jjrm\lri\mrk"S+S,\n\ 5ro"S-S.5rp\"S/S055rq\oR\q"S1S2S3S4S S59\oR\q"S1S6S S79\oR\q"S8S2S9S:S S59\oR\q"S8S6S S79\oR\q"S;S\yS?'\R\oR4SS@jj5r|SSSSB.SSCjjjr}SSDjr~SSEjrSSFjrSSGjrSSHjrSSIjrSSSJjjrSSKjr\/"SLSM9SSSNjj5rSSSOjjrSSSPjjrSSQjrSRrSSSjrSSTjr\i"SUSVSWSX9SSSYjj5r\k"SU5SSSZjj5rSSS\jjrSS]jrSSS^jjrSS_jrSSS`jjrSSajrSSbSc.SSdjjjrSSejrSSbSc.SSfjjjrSSgjrSSbSAS SSSSh.SSijjjrSSjjrSSSkjjrSSSljjrSSmjrSSSnjjrSSSojjrSSpjrSqrSSrjrSSSsjjr"StSu\GRHGRJ5rSSvjrSSwjrSSxjrSSyjrSSzjr\0SS{j5r\0SS|j5r"S}S~\GRHGRJ5r"SS\GRHGRJ5r"SS\GRHGRJ5r"SS\GRHGRJ5rSSSjjrSSS.SSjjjr\pGRi\oGRj\\\/S[S9SSSjj5r\pGRi\oR\\\/S[S9SSSjj5r\pGRi\oR\\\/SS9SSSjj5r\pGRi\oGRr\\\/S9SSSjj5r\pGRi\oGRv\\\/S9SSSjj5r\pGRi\oR\\\/S[S9SSSjj5r\pGRi\oR\\\/SS9SSSjj5r\pGRi\oR\\\/SS9SSSjj5r\pGRi\oGR\\\/S9SSSjj5r\pGRi\oGR\\\/S9SSSjj5r\pGRi\oGR\~\\/S9SSSjj5rSSSjjr\pGRi\oR\\/S[S9SSSjj5r\pGRi\oGR\\\/S[S9SSSjj5r\pGRi\oGR\\/S9SSSjj5r\pGRi\oGR\\\/S[S9SSSjj5r\pGRi\oGR\\/S9SSSjj5r\pGRi\oGR\\\/S[S9SSSjj5r\pGRi\oGR\\/S9SSSjj5r\pGRi\oGR\\\/S[S9SSSjj5r\pGRi\oR\\\/S[S9SSSjj5r\pGRi\oGR\\\/S9SSSjj5r\pGRi\oGR\"SS5\/S9SSSjj5r\pGRi\oGR\"SS5\/S9SSSjj5r\pGRi\oGR\"SS5\/S9SSSjj5r\pGRi\oGR\"SS5\/S9SSSjj5r\pGRi\oGR\~\\/S9SSSjj5rg!\J\K\L4a*rM\;RS\MS35 SrSrSSrMCMG NSrMCMff=f!\J\K\L4a.rM\;RS\MS35 Sr?SrUSrVSrWSrXSrYSrZSrMCMG NSrMCMff=f!\J\K\L4a$rM\;RS \MS35 Sr@Sr]SrMCMG N,SrMCMff=f!\J\K\L4a$rM\;RS"\MS35 SrASr_SrMCMG NKSrMCMff=f!\J\K\L4a$rM\;RS$\MS35 SrBSrbSrMCMG NjSrMCMff=f!\J\K\L4a$rM\;RS%\MS35 SrCSreSrMCMG NSrMCMff=f)) annotationsN) dataclass)Enum) TYPE_CHECKINGAnyCallable) get_loggeris_aiter_availableis_aiter_versionis_flash_attn_3_availableis_flash_attn_availableis_flash_attn_versionis_kernels_availableis_kernels_versionis_sageattention_availableis_sageattention_versionis_torch_npu_availableis_torch_versionis_torch_xla_availableis_torch_xla_versionis_xformers_availableis_xformers_version)DIFFUSERS_ATTN_BACKENDDIFFUSERS_ATTN_CHECKS)lru_cache_unless_exportmaybe_allow_in_graph)gather_size_by_comm)ParallelConfigz2.6.3z0.1.5z2.1.1z2.5.0z2.2z0.0.29>=)flash_attn_funcflash_attn_varlen_func)_wrapped_flash_attn_backward_wrapped_flash_attn_forwardz.flash_attn is installed but failed to import: z+. Falling back to native PyTorch attention.F)r")r#zflash_attn_3 failed to import: z#. Falling back to native attention.zaiter failed to import: )sageattnsageattn_qk_int8_pv_fp8_cuda!sageattn_qk_int8_pv_fp8_cuda_sm90sageattn_qk_int8_pv_fp16_cudasageattn_qk_int8_pv_fp16_tritonsageattn_varlenz sageattention failed to import: z!flex_attention failed to import: )npu_fusion_attentionztorch_npu failed to import: )flash_attentionztorch_xla failed to import: zxformers failed to import: z2.4.0) device_typesschemacSnUcU$U$)NcU$Nfuncs ]/home/wildlama/miniconda3/lib/python3.13/site-packages/diffusers/models/attention_dispatch.pywrapcustom_op_no_op..wrapKr3)namefn mutates_argsr.r/r7s r6custom_op_no_opr> zt)r)r:)lib _stacklevelcSnUcU$U$)NcU$r2r3r4s r6r7!register_fake_no_op..wrapr9r:r3)opr<r@rAr7s r6register_fake_no_oprFr?r:c|\rSrSrSrSrSrSrSrSr Sr S r S r S r S rS rSrSrSrSrSrSrSrSrSrSrSrSrSrSrSrg)AttentionBackendNameflash flash_hub flash_varlenflash_varlen_hub flash_4_hub_flash_3_flash_varlen_3 _flash_3_hub_flash_3_varlen_hubaiterflexnative _native_cudnn_native_efficient _native_flash _native_math _native_npu _native_xlasagesage_hub sage_varlen_sage_qk_int8_pv_fp8_cuda_sage_qk_int8_pv_fp8_cuda_sm90_sage_qk_int8_pv_fp16_cuda_sage_qk_int8_pv_fp16_tritonxformersr3N)__name__ __module__ __qualname____firstlineno__FLASH FLASH_HUB FLASH_VARLENFLASH_VARLEN_HUB FLASH_4_HUB_FLASH_3_FLASH_VARLEN_3 _FLASH_3_HUB_FLASH_3_VARLEN_HUBAITERFLEXNATIVE _NATIVE_CUDNN_NATIVE_EFFICIENT _NATIVE_FLASH _NATIVE_MATH _NATIVE_NPU _NATIVE_XLASAGESAGE_HUB SAGE_VARLEN_SAGE_QK_INT8_PV_FP8_CUDA_SAGE_QK_INT8_PV_FP8_CUDA_SM90_SAGE_QK_INT8_PV_FP16_CUDA_SAGE_QK_INT8_PV_FP16_TRITONXFORMERS__static_attributes__r3r:r6rHrHs EI!L)KH'O!L/ E D F#M+#M!LKK DHK ;%E"!=#A Hr:rHc\rSrSr0r0r0r\"5r\ "\ 5r \ r \S S Sjj5r\S5r\S Sj5r\S5r\S Sj5rSrg) _AttentionBackendRegistryi NcV^^^^[RSTST35 UUUU4SjnU$)NzRegistering attention backend: z with constraints: c6>UTRT'T=(d /TRT'[[R"U5R R 55TRT'T(a%TRRTR5 U$r2) _backends _constraintssetinspect signature parameterskeys_supported_arg_names_supports_context_paralleladdvalue)r5backendcls constraintssupports_context_parallels r6 decorator5_AttentionBackendRegistry.register..decoratorsu%)CMM' "(3(9rC  W %03G4E4Ed4K4V4V4[4[4]0^C $ $W -(..227==AKr:)loggerdebug)rrrrrs```` r6register"_AttentionBackendRegistry.registers4  6wi?RS^R_`a  r:cLURURUR4$r2)_active_backendrrs r6get_active_backend,_AttentionBackendRegistry.get_active_backend's"""CMM#2E2E$FFFr:cXlgr2)r)rrs r6set_active_backend,_AttentionBackendRegistry.set_active_backend+s%r:cH[URR55$r2)listrrrs r6 list_backends'_AttentionBackendRegistry.list_backends/sCMM&&())r:c8URUR;nU$r2)rr)rrrs r6_is_context_parallel_available8_AttentionBackendRegistry._is_context_parallel_available3s %,MMS5S5S$S!((r:r3)NF)rrHrzlist[Callable] | Nonerbool)rstr)rrHreturnr)rdrerfrgrrrrrrHrrr_checks_enabled classmethodrrrrrrr3r:r6rr sIL!$*+ABO+O.2*/ %+$( &GG&&**)%) ))r:rc\rSrSr%SrS\S'S\S'SrS\S'SrS \S 'SrS \S 'Sr S\S 'Sr S\S'Sr S \S'Sr S \S'Sr g)_HubKernelConfigi<zEConfiguration for downloading and using a hub-based attention kernel.rrepo_id function_attrNz str | Nonerevisionz int | NoneversionzCallable | None kernel_fnwrapped_forward_attrwrapped_backward_attrwrapped_forward_fnwrapped_backward_fnr3)rdrerfrg__doc____annotations__rrrrrrrrr3r:r6rr<sZO LHjGZ!%I%'+*+(,:,*..+//r:rzkernels-community/flash-attn3r"z(flash_attn_interface._flash_attn_forwardz)flash_attn_interface._flash_attn_backward)rrrrrr#)rrrzkernels-community/flash-attn2z0flash_attn_interface._wrapped_flash_attn_forwardz1flash_attn_interface._wrapped_flash_attn_backwardz kernels-community/sage-attentionr&zkernels-staging/flash-attn4z.dict['AttentionBackendName', _HubKernelConfig]_HUB_KERNELS_REGISTRYc#F# U[R;a[SUS35e[U5n[ U5 [ U5 [R n[RU5 Sv [RU5 g![RU5 f=f7f)z6 Context manager to set the active attention backend. zBackend z is not registered.N)rr ValueErrorrH%_check_attention_backend_requirements"_maybe_download_kernel_for_backendrr)r old_backends r6attention_backendrrs /9998G9,?@AA"7+G)'2&w/+;;K009B !44[A!44[AsA*B!-B1B!BB!)rparallel_configc U=(d 0nU c[R5upO*[U 5n [RR U 5n UUUUUUUS.UESU 0En [ (aX}S'[R (ay[U 5[[RU 5- nU(a[RSU SUS35 [RR U 5H nU"S0U D6 M U R5VVs0sH"unnU[RU ;dMUU_M$ n nnU "S0U D6$s snnf)N)querykeyr attn_mask dropout_p is_causalscale_parallel_config enable_gqaz5Removing unsupported arguments for attention backend z: .r3) rrrHrget_CAN_USE_FLEX_ATTNrrrrwarningritems)rrrrrrrrattention_kwargsrr backend_name backend_fnkwargsremoved_kwargscheckkvs r6dispatch_attention_fnrsN(-2$=#O#O#Q j+G4 .88<<\J     O F)| 00Vs+D+Y+YZf+g'hh  NNRS_R``bcqbrrst u.;;?? ME OFON &||~ s~tq!6O6d6deq6r1rdad~F s   ts E2Ec 0UbU(a [S5egg)Nz8`is_causal` cannot be True when `attn_mask` is not None.)r)rrrs r6_check_attn_mask_or_causalrsSTT"+r:c URUR:wdURUR:wa [S5eURUR:wdURUR:wa [S5eg)Nz1Query, key, and value must be on the same device.z/Query, key, and value must have the same dtype.)devicerdtyperrrrs r6 _check_devicers] ||szz!U\\U\\%ALMM {{cii5;;%++#=JKK$>r:c f[XU5 URRS:wa [S5eg)Ncudaz/Query, key, and value must be on a CUDA device.)rrtyperrs r6_check_device_cudars/%e$ ||F"JKK#r:c^^SUU4SjjnU$)Nc>[XU5 [RRUR5TT4:a[ STSTS35eg)NzJQuery, key, and value must be on a CUDA device with compute capability >= r)rtorchrget_device_capabilityrr)rrrrmajorminors r6check_device_cuda:_check_device_cuda_atleast_smXY..check_device_cudasW5u- :: + +ELL 9UEN J\]b\ccdejdkklm  Kr:r torch.TensorrrrrrNoner3)rrrs`` r6_check_device_cuda_atleast_smXYrs r:c URUR:wa [S5eURUR:wa [S5eg)Nz'Query and key must have the same dtype.z)Query and value must have the same dtype.)rrrs r6_check_qkv_dtype_matchrs? {{ciiBCC {{ekk!DEE"r:c [XU5 UR[R[R4;a [ S5eg)Nz9Query, key, and value must be either bfloat16 or float16.)rrrbfloat16float16rrs r6_check_qkv_dtype_bf16_or_fp16rs65u- {{5>>5==99TUU:r:c URSURS:wa [S5eURSURS:wa [S5eUb,URSURS:wa [S5egg)Nz0Query and key must have the same head dimension.z1Key and value must have the same sequence length.z4Attention mask must match the key's sequence length.)shaper)rrrrrs r6 _check_shapers {{2#))B-'KLL yy} B'LMM!4 " !EOPP"Fr:cU[R[R4;a,[(d [ SUR S[ S35egU[R[R4;a%[(d[ SUR S35egU[R[R[R[R[R[R4;a[!5(d[ SUR S35e[#SS 5(d[ SUR S 35eU[R:Xa+[#SS 5(d[ SUR S 35eggU[R$:Xa,[&(d [ S UR S[(S35egU[R*[R,[R.[R0[R2[R44;a,[6(d [ SUR S[8S35egU[R::Xa%[<(d[ SUR S35egU[R>:Xa%[@(d[ SUR S35egU[RB:Xa,[D(d [ SUR S[FS35egU[RH:Xa,[J(d [ SUR S[LS35egg)NzFlash Attention backend 'zb' is not usable because of missing package or the version is too old. Please install `flash-attn>=z`.zFlash Attention 3 backend 'zp' is not usable because of missing package or the version is too old. Please build FA3 beta release from source.z Backend 'zl' is not usable because the `kernels` package isn't available. Please install it with `pip install kernels`.r!z0.12zj' needs to be used with a `kernels` version of at least 0.12. Please update with `pip install -U kernels`.z0.12.3zl' needs to be used with a `kernels` version of at least 0.12.3. Please update with `pip install -U kernels`.zAiter Attention backend 'z]' is not usable because of missing package or the version is too old. Please install `aiter>=zSage Attention backend 'ze' is not usable because of missing package or the version is too old. Please install `sageattention>=zFlex Attention backend 'zd' is not usable because of missing package or the version is too old. Please install `torch>=2.5.0`.zNPU Attention backend 'za' is not usable because of missing package or the version is too old. Please install `torch_npu`.zXLA Attention backend 'za' is not usable because of missing package or the version is too old. Please install `torch_xla>=zXformers Attention backend 'z`' is not usable because of missing package or the version is too old. Please install `xformers>=)'rHrhrj_CAN_USE_FLASH_ATTN RuntimeErrorr_REQUIRED_FLASH_VERSIONrmrn_CAN_USE_FLASH_ATTN_3rirkrorpr{rlrrrq_CAN_USE_AITER_ATTN_REQUIRED_AITER_VERSIONrzr|r}r~rr_CAN_USE_SAGE_ATTN_REQUIRED_SAGE_VERSIONrrrrx_CAN_USE_NPU_ATTNry_CAN_USE_XLA_ATTN_REQUIRED_XLA_VERSIONr_CAN_USE_XFORMERS_ATTN_REQUIRED_XFORMERS_VERSION)rs r6rrs'--/C/P/PQQ""+GMM?;]^u]vvxy # )224H4X4XY Y$$-gmm_=mn % &&--))00%%((  $%%GMM?+WX "$//GMM?+UV  *66 6?QRVX`?a?aGMM?+WX @b 6 (.. .""+GMM?;XYpXqqst # !!((66;;7799  "!*7==/:_`v_wwyz " (-- -!!*7==/:^_ " (44 4  )'--9Z[ ! (44 4  )'--9Z[pZqqst ! (11 1%%.w}}o>^_y^zz|} & 2r:)maxsizec[R"U4U[RUS9n[R"U4U[RUS9n[R"US-[RUS9n[R"US-[RUS9n[R"USS9USS&[R"USS9USS&UR 5R 5nUR 5R 5n XE4Xg4X44$)Nrrrrdim)rfullint32zeroscumsummaxitem) batch_size seq_len_q seq_len_kvr seqlens_q seqlens_k cu_seqlens_q cu_seqlens_k max_seqlen_q max_seqlen_ks r63_prepare_for_flash_attn_or_sage_varlen_without_maskr"Ps J=)5;;vVI J=*EKKPVWI;;zA~U[[PL;;zA~U[[PL||I15L||I15L==?'')L==?'')L  !L#?,A] ]]r:c[R"U4U[RUS9nURS[RS9n[R"US-[RUS9n[R"US-[RUS9n[R "USS9USS&[R "USS9USS&UR 5R5nUR 5R5n XE4Xg4X44$)Nrr)rrrr)rrrsumrrrr) rrrrrrrrr r!s r60_prepare_for_flash_attn_or_sage_varlen_with_maskr%bs  J=)5;;vVI !5;; 7I;;zA~U[[PL;;zA~U[[PL||I15L||I15L==?'')L==?'')L  !L#?,A] ]]r:c8Uc [XX$5$[XX45$r2)r"r%)rrrrrs r6&_prepare_for_flash_attn_or_sage_varlenr'ss&B:Zdmm ;JS\ eer:c UR[R:wa[SURS35eURS:Xa"UR S5R X5nGOAURS:XaHURS5SU4;a[SURSSUS35eUR X5nOURS :XaWURS5SU4;a[SURSSUS 35eURSS 9nUR X5nOURS :XaZURS5SU4;a[SURSSUS 35eUR USSU5nURSS 9nO[SUR35eURX4:wa[SURSUSUS35eU$)z Normalize an attention mask to shape [batch_size, seq_len_k] (bool) suitable for inferring seqlens_[q|k] in FlashAttention/Sage varlen. Supports 1D to 4D shapes and common broadcasting patterns. z)Attention mask must be of type bool, got rrrr zattn_mask.shape[0] (z) must be 1 or z for 2D attention mask.z for 3D attention mask.rz for 4D attention mask.r)rr z"Unsupported attention mask shape: z.Normalized attention mask shape mismatch: got z , expected (z, )) rrrrndim unsqueezeexpandsizerany)rr seq_len_ks r6_normalize_attn_maskr2s%**$DY__DUUVWXX~~''*11*H 1  >>! Q O 3&yq'9&:/*Ulm $$Z; 1  >>! Q O 3&yq'9&:/*Ulm MMaM( $$Z; 1  >>! Q O 3&yq'9&:/*Ulm $$ZRC MMfM- =ioo=NOPP:11>FOOU[UcUcd 3MCWCWXF  (<]LgLg(hF % !)=mMiMi)jF & "  @@PP`ab`cde sBD E&EEz,_diffusers_flash_attn_3::_flash_attn_forwardr3r)r=r.cSn[S0SU_SU_SU_SU_SU_SU_SU_S U_S U_S U_S U _S U _SU _SU _SU _SU_SS_6nUtnnnURSSS5nUU4$)Nrrqrr softmax_scalecausalqv q_descale k_descale v_descale window_sizeattention_chunksoftcap num_splitspack_gqa deterministic sm_marginreturn_attn_probsTrr rr3)flash_attn_3_funcpermute)rPrrrQrRrSrTrUrVrXrYrZr[r\r]rWresultoutlse_s r6_wrapped_flash_attn_3res&K    $      ($ !"#F&LCq ++aA C 8Or:cSnURunnnnUUU4n[R"U5URU54$)NrO)rr empty_like new_empty)rPrrrQrRrSrTrUrVrXrYrZr[r\r]rWrseq_len num_headshead_dim lse_shapes r6rdrdsH$K01ww,JHWi0I   A  I 6 66r:Tc 8U (a [S5eU (a0URXU5 X@lXPlX`lXplXlSXU45upn[RRRUUUUUUUUS9n U RSSSS5n U $)Nz1Native attention does not support return_lse=Truec3H# UHoRSSSS5v M g7frr rr)Nr`.0xs r6 /_native_attention_forward_op..G"L8K11aA..8K "rrrrrrrrrr rr)) rsave_for_backwardrrrrrrnn functionalscaled_dot_product_attentionr` ctxrrrrrrrr return_lse _save_ctxrrbs r6_native_attention_forward_opr,sLMM e%0! ! !  #LU8KLE ((   : :  ; C ++aAq !C Jr:c URupEnURS5 URS5 URS5 [R"5 SXEU45upxn [RR R UUU URURURURURS9n U RSSSS5n [RRXX/USS 9upn SSS5 W RSSSS5nW RSSSS5nW RSSSS5nXU4$!,(df  NN=f) NTc3H# UHoRSSSS5v M g7frorprqs r6rt0_native_attention_backward_op..ds""VBUQ99Q1a#8#8BUrwrxrr rr)F)outputsinputs grad_outputs retain_graph) saved_tensorsrequires_grad_r enable_gradrzr{r|rrrrrr`autogradgrad)r~grad_outargsrrrrquery_tkey_tvalue_trb grad_query_t grad_key_t grad_value_t grad_querygrad_key grad_values r6_native_attention_backward_oprWsH ))E t     "V5uBU"Vhh!!>>mmmmmm))~~?  kk!Q1%161D1D% 9_d2E2 . , $%%aAq1J!!!Q1-H%%aAq1J  ++-  s B'E Ec U(a [S5eSn URSS5R5nURSS5R5nURSS5R5nXX#4- n [RR R UUUUU UUSUS9 u pnnnnnnnXXUUU4- n U (a5UR"U 6 XPlX`l Xpl X@l UUl UUl U RSS5R5n Ub URSS5R5nU (aX4$U $)Nz6`enable_gqa` is not yet supported for cuDNN attention.r3rr F) rrr attn_biascompute_log_sumexprrreturn_debug_maskr)r transpose contiguousropsaten#_scaled_dot_product_cudnn_attentionryrrrrmax_qmax_kr~rrrrrrrrrrrtensors_to_saverbrc cum_seq_q cum_seq_krr philox_seed philox_offsetdebug_attn_masks r6_cudnn_attention_forward_oprsNQRRO OOAq ! , , .E --1  ( ( *C OOAq ! , , .Es**O  ::)# ; ^CiE5+}oSY ]SSO /! !  !   --1  ( ( *C mmAq!,,.#C:,,r:cURu pEpgppn URSS5R5nURSS5R5nURSS5R5n[RR R UUUUUUU U URU U URURURURURS9upnSXU45upnXU4$)Nrr ) logsumexprrrrrrrrrrc3`# UH$oRSS5R5v M& g7frr Nrrrqs r6rt/_cudnn_attention_backward_op..''sPr1 Aq(9(D(D(F(FPr,.) rrrrrr,_scaled_dot_product_cudnn_attention_backwardrrrrrrr~rrrrrrrbrcrrrrrrrs r6_cudnn_attention_backward_oprs UXTeTeQECIM!!!Q'224H --1  ( ( *C OOAq ! , , .E(-yy~~'b'b    #--iiii----ii!(c($J*$(tQ[gqPr's$J*  ++r:c ~U(a [S5eSn URSS5R5nURSS5R5nURSS5R5nXX#4- n [RR R UUUUUSUS9u pnnnnnnnXXUUU4- n U (a/UR"U 6 XPlX`l Xpl UUl UUl U RSS5R5n Ub URSS5R5nU (aX4$U $)Nz=`enable_gqa` is not yet supported for native flash attention.r3rr F)rrrrrrr) rrrrrr#_scaled_dot_product_flash_attentionryrrrrrrs r6"_native_flash_attention_forward_oprsAXYYO OOAq ! , , .E --1  ( ( *C OOAq ! , , .Es**O  ::# ; ^CiE5+}oSY ]SSO /! !    --1  ( ( *C mmAq!,,.#C:,,r:cURu pEpgppn URSS5R5nURSS5R5nURSS5R5n[RR R UUUUUUU U U U URURURURURS9upnSXU45upnXU4$)Nrr ) rrrrrrrrrrc3`# UH$oRSS5R5v M& g7frrrqs r6rt6_native_flash_attention_backward_op..3rr) rrrrrr,_scaled_dot_product_flash_attention_backwardrrrrrrs r6#_native_flash_attention_backward_oprs UXTeTeQECIM!!!Q'224H --1  ( ( *C OOAq ! , , .E',yy~~'b'b    #iiii----ii(c($J*"(tQ[gqPr's$J*  ++r:rOrWc hUb [S5eU(a [S5eSn SnSn[SXU455nUcURSS-nU(dU b$U RRS:a US :aUOS n[ R "U5 [UUUUUUU S U SU UU 5 unnnnURS S S5nSSS5 U (a?URXUWWW5 XPl Xpl X`l Xl XlXlXlU (aWW4$W$!,(df  Na=f) Nz2`attn_mask` is not yet supported for flash-attn 2.z3`enable_gqa` is not yet supported for flash-attn 2.rFc38# UHoRv M g7fr2 requires_gradrqs r6rt._flash_attention_forward_op..QD0C10CrrrKH9r )rr0rcontext_parallel_config _world_sizerset_grad_enabledr%r`ryrrrrWrY alibi_slopesr\)r~rrrrrrrrrrrrWrYrr\ grad_enabledrbrcS_dmask rng_states r6_flash_attention_forward_opr9sJ MNNNOOGLMDU0CDDL } BD)(49I9a9a9m9mpq9q!*QIE    -'B       N N    ( $S'9kk!Q" .  e%c9E!  ! % ')#C:,,5 . -s 4D## D1cURupEpgp[R"U5[R"U5[R"U5pn [UUUUUUU U U URUR UR URSURSURURURU 5n U SSURS24n U SSURS24n U SSURS24n XU 4$)Nrr.r) rrrgr$rrrrWrYrr\r)r~rrrrrrrbrcrrrrlse_ds r6_flash_attention_backward_oprws .1->->*EC','7'7'>@P@PQT@UW\WgWghmWn*J (             % E,C!58>>"#5!556J1x~~b1112HC!58>>"#5!556J  ++r:c Ub [S5eU(a [S5e[[Rn U RnU R nUbUc [ S5eUcURSS-nSnSnSn[SXU455nU(dU b$U RRS :a US :aUOS n[R"U5 U"UUUUUUU S U S UUU 5 unnnnURS S S 5R5nSSS5 U (aBURXUWWW5 XPlXplX`lXlUUlUUlUUlU (aWW4$W$!,(df  Nd=f) Nz<`attn_mask` is not yet supported for flash-attn hub kernels.z=`enable_gqa` is not yet supported for flash-attn hub kernels.zFlash attention hub kernels must expose `_wrapped_flash_attn_forward` and `_wrapped_flash_attn_backward` for context parallel execution.rrrFc38# UHoRv M g7fr2rrqs r6rt2_flash_attention_hub_forward_op..rrrrrr )rrrHrirrrrr0rrrrr`rryrrrrWrYrr\)r~rrrrrrrrrrrrWrHrrrYrr\rrbrcrrs r6_flash_attention_hub_forward_oprs WXXXYY "#7#A#A BF22 44!%8%@ .   } BD)GLMDU0CDDL(49I9a9a9m9mpq9q!*QIE    -'9       N N    ( $S'9kk!Q"--/ .  e%c9E!  ! % ')#C:,,5 . -s ?E11 E?cp[[RnURnUc [ S5eUR upgpp[ R"U5[ R"U5[ R"U5pn U"UUUUU U U U UURURURURSURSURURURU 5nU SSURS24n U SSURS24n USSURS24nXU4$)NzfFlash attention hub kernels must expose `_wrapped_flash_attn_backward` for context parallel execution.rr.r)rrHrirrrrrgrrrrWrYrr\r)r~rrrrHrrrrrbrcrrrrrds r6 _flash_attention_hub_backward_oprsO ##7#A#A BF 44" t  .1->->*EC','7'7'>@P@PQT@UW\WgWghmWn*J             % A*C!58>>"#5!556J1x~~b1112HC!58>>"#5!556J  ++r:rWrYrZr[r\r]c @Ub [S5eUS:wa [S5eU(a [S5e[[RnURnUc [ S5eUcUR SS-nU"UUUSSSSSSSSSSSSSSSSSSSSU4UU SU S SU UUUS .6tnnnU (a!URSS S 5R5OSnU (a:URXUUU5 Xpl X`l Xl Xl UUlUUlU (aUU4$U$) Nz>`attn_mask` is not yet supported for flash-attn 3 hub kernels.rz>`dropout_p` is not yet supported for flash-attn 3 hub kernels.z?`enable_gqa` is not yet supported for flash-attn 3 hub kernels.ztFlash attention 3 hub kernels must expose `flash_attn_interface._flash_attn_forward` for context parallel execution.rrrr)rRwindow_size_leftwindow_size_rightrXrYrZr[r]r )rrrHrorrrr`rryrrrWrYr\r])r~rrrrrrrrrrrrWrYrZr[r\r]rHrrb softmax_lserdrcs r6!_flash_attention_3_hub_forward_oprsx*YZZCYZZZ[[ "#7#D#D EF22! .   } BD)-                        1!2$Q%a.A!CqF8B+  aA & 1 1 3tC e%kB ! % )! #C:,,r:cf[[RnURnUc [ S5eUR upgpn [ R"U5n [ R"U5n [ R"U5n U"UUUUU U SSSSSSU U U URURURSURSURURUR5 U SSURS24n U SSURS24n U SSURS24n XU 4$)NzuFlash attention 3 hub kernels must expose `flash_attn_interface._flash_attn_backward` for context parallel execution.rr.r)rrHrorrrrrgrrrWrYr\r]r)r~rrrrHrrrrrbrrrrs r6"_flash_attention_3_hub_backward_oprgsV ##7#D#D EF 44" .  +.*;*;'EK!!%(J$H!!%(J                 -2C!58>>"#5!556J1x~~b1112HC!58>>"#5!556J  ++r:c Ub [S5eUS:a [S5eU(a [S5e[UUUSUUU S9n Sn U (aU tpnU RSSS 5n U (aX4$U $ Nz4`attn_mask` is not yet supported for Sage attention.rz4`dropout_p` is not yet supported for Sage attention.z5`enable_gqa` is not yet supported for Sage attention.NHDrPrr tensor_layoutrsm_scalerrr r)rr&r`)r~rrrrrrrrrrrrbrcrds r6_sage_attention_forward_oprsOPP3OPPPQQ    C C 1kk!Q"#C:,,r:c 2Ub [S5eUS:a [S5eU(a [S5e[[RRn U "UUUSUUU S9n SnU (a%U tpnUR SSS 5R 5nU (aX4$U $r)rrrHr{rr`r)r~rrrrrrrrrrrr5rbrcrds r6_sage_attention_hub_forward_oprsOPP3OPPPQQ !5!>!> ? I ID    C C 1kk!Q"--/#C:,,r:c[S5e)Nz4Backward pass is not implemented for Sage attention.NotImplementedErrorr~rrs r6_sage_attention_backward_oprs T UUr:cUb [R"US:g5(aSnUGb-URS:XaURSURS:XaURSURS:XakURSURSURSpTnUR S5R X4U5R S5R 5nORURS:XaBURSSS:Xa/UR SSURSS5R 5nUR[R5)nU$)Nrr rr*r))rrr) rallr,rr-r.rtor)rrrrrrs r6_maybe_modify_attn_mask_npurs%9>!:!:  >>Q 9??1#5Q#GIOO\]L^bebkbklmbnLn090BEKKPQNTWT]T]^_T`:J!++A.55jZXbbcdepprI ^^q Y__Qq%9V%C!((RQDOOQI\\%**-- r:c U (a [S5e[XU5n[UUUURS5USSUc&S[R "UR S5- OUSSSU- SSS 9 Sn U $) NANPU attention backend does not support setting `return_lse=True`.r BSND?rFr atten_mask input_layoutpser pre_tockens next_tockens keep_probsync inner_precise)rrr,r/mathsqrtrr}s r6_npu_attention_forward_oprs\]]+E BI     1  27-cDIIekk"o..U /   C Jr:c[S5e)Nz:Backward pass is not implemented for Npu Fusion Attention.r)r~rrrs r6_npu_attention_backward_opr*s Z [[r:cd[U[R5(aUR5nU$r2) isinstancefuncolAsyncCollectiveTensorwait)tensors r6 _wait_tensorr:s%&&6677 Mr:cURnUR5n[R"USSU5nUR U5n[ U5nU$r2)rflattenr all_to_all_singlereshaper)rsgrouprs r6_all_to_all_singler@sJ GGE A  D$6A %AQA Hr:c[RRU5nUS:XaUS:XaURupVpxXd-n Xt-n UR XVXJU5R SS5R 5n US:a [XS9n OU n U R XX5RSSSS5R 5n U R XYX5n U $US:XaUS:XaURupYpX-nX-nUR XTXjU5RSSSSS5R XJXeU5n US:a [X5n OU n U R XvXX5R SS5R 5n U R XVXx5n U $[S5e)u Perform dimension sharding / reassembly across processes using _all_to_all_single. This utility reshapes and redistributes tensor `x` across the given process group, across sequence dimension or head dimension flexibly by accepting scatter_idx and gather_idx. Args: x (torch.Tensor): Input tensor. Expected shapes: - When scatter_idx=2, gather_idx=1: (batch_size, seq_len_local, num_heads, head_dim) - When scatter_idx=1, gather_idx=2: (batch_size, seq_len, num_heads_local, head_dim) scatter_idx (int) : Dimension along which the tensor is partitioned before all-to-all. gather_idx (int): Dimension along which the output is reassembled after all-to-all. group : Distributed process group for the Ulysses group. Returns: torch.Tensor: Tensor with globally exchanged dimensions. - For (scatter_idx=2 → gather_idx=1): (batch_size, seq_len, num_heads_local, head_dim) - For (scatter_idx=1 → gather_idx=2): (batch_size, seq_len_local, num_heads, head_dim) r rrrr)r*zaJ!O:;6 92#7 IIj1AT\ ] Yq!_ Z\  a $V9CCkk'IQQRSUVXY[\]hhjkk*I  jAo:;6 _#6 3  IIjMT\ ] WQ1a # W% S[ \  a '6FF *OYYZ[]^_jjl 9O WXXr:c<\rSrSrSr\SSj5r\S5rSrg)SeqAllToAllDimiz all_to_all operation for unified sequence parallelism. uses _all_to_all_dim_exchange, see _all_to_all_dim_exchange for more info. c>XlX0lX@l[X#XA5$r2)r scatter_id gather_idr )r~rinputr$r%s r6forwardSeqAllToAllDim.forwards #! '9LLr:cz[RURUURUR5nSUSS4$r2)r"applyrr%r$)r~r grad_inputs r6backwardSeqAllToAllDim.backwards;#)) II  MM NN  j$--r:r3N)r r) rdrerfrgr staticmethodr'r,rr3r:r6r"r"s1 MM ..r:r"c[R"US9nSnX-S:waIX1U-- nX-U-nXE:d SUSU35e[R"USSSU45R 5nX4$)zMaybe pad the head dimension to be divisible by world_size. x: torch.Tensor, shape (B, S_LOCAL, H, D) H: int, original global head num return: tuple[torch.Tensor, int], padded tensor (B, S_LOCAL, H + H_PAD, D) and H_PAD rrPadding head num ( should be less than new local head num )distrFpadr)rsHr world_sizeH_PAD NEW_H_LOCALs r6_maybe_pad_qkv_headr9s $$51J E~*n-yZ/ "t&7w>fgrfs$tt" EE!aAu% & 1 1 3 8Or:c[R"US9n[R"US9nUS:aX4S- :XaUSS2SS2SU*2SS24nUR5$)zMaybe unpad the head dimension. x: torch.Tensor, shape (B, S_GLOBAL, H_LOCAL + H_PAD, D) H_PAD: int, head padding num return: torch.Tensor, unpadded tensor (B, S_GLOBAL, H_LOCAL, D) rrrN)r2get_rankrr)rsr7rrankr6s r6_maybe_unpad_qkv_headr=sW ==u %D$$51J qyT!^+ aGeVGQ  <<>r:cUcUS4$[R"US9n[R"US9nSnX-S:waQXAU-- nX-U-nXV:d SUSU35eX4S- :Xa)[R"USSSU45R 5nX4$)zMaybe pad the head dimension to be divisible by world_size. x: torch.Tensor, shape (B, S_GLOBAL, H_LOCAL, D) H: int, original global head num return: tuple[torch.Tensor, int], padded tensor (B, S_GLOBAL, H_LOCAL + H_PAD, D) and H_PAD rrr0r1r)r2r;rr3r4r)rsr5rr<r6r7r8s r6_maybe_pad_o_headr?s  y!t ==u %D$$51J E~*n-yZ/ "t&7w>fgrfs$tt" > !a!Q5)*557A 8Or:cRUS:aUSS2SS2SU*2SS24nUR5$)zMaybe unpad the head dimension. x: torch.Tensor, shape (B, S_LOCAL, H_GLOBAL + H_PAD, D) H_PAD: int, head padding num return: torch.Tensor, unpadded tensor (B, S_LOCAL, H_GLOBAL, D) rN)r)rsr7rs r6_maybe_unpad_o_headrAs0  qy aGeVGQ  <<>r:c [UR5S:XdS5e0nURSUS'URSUS'U$)Nr*zEQuery tensor must be 4-dimensional of shape (B, S_LOCAL, H_GLOBAL, D)r NUM_QO_HEADr Q_S_LOCAL)lenr)rr extra_kwargss r6ulysses_anything_metadatarGsN u{{ q i"ii L"'++a.L % AL r:c v^^^ [R"TS9nTRupEpg[TUT5umm UT -U-nTR XEX8U5R SSSSS5R 5mU/U-n [UT5n TRSS5m[R"TXT5mSU UU4Sjjn U $) zc x: torch.Tensor, shape (B, S_LOCAL, H, D) return: Callable that returns (B, S_GLOBAL, H_LOCAL, D) rr rrr)r*c|>[T5mTRSSSS5R5m[TTT5mT$)Nrrr r))rr`rr=)r7rrssr6r -all_to_all_single_any_qkv_async..waits> O IIaAq ! , , . !!UE 2r:rr) r2rrr9rr`rrrr r) rsrrr6BS_LOCALr5DH_LOCALinput_split_sizesoutput_split_sizesr r7s `` @r6all_to_all_single_any_qkv_asyncrRs$$51JwwA"1a/HAu5yZ'G !j15==aAq!LWWYA J.-We< !QA  $65QA Kr:c ^^^ ^ ^ ^ ^ ^URSS5n[R"TS9m[TUT5umm TRnUum nm m URS5m [ T T5nTR SSSS5R5mT /T-n[R"TXvT5mS U U U U U UUU4S jjnU$) zj x: torch.Tensor, shape (B, S_GLOBAL, H_LOCAL, D) return: Callable that returns (B, S_LOCAL, H_GLOBAL, D) rCNrrDrrr r)c>[T5mTRTTTTT5mTRSSSSS5R5mTRTTTT-T5m[ TTT5mT$)Nr rrr)r*)rrr`rrA)rLrNrOr7rMrr6rssr6r +all_to_all_single_any_o_async..wait6sm O IIj'1gq 9 IIaAq! $ / / 1 IIa*w"6 : 5% 0r:rK) rr2rr?rrr`rr r)rsrrr5rS_GLOBALrPrQr rLrNrOr7rMr6s`` @@@@@@r6all_to_all_single_any_o_asyncrWs  =$'A$$51J Au-HAu GGE %Q'1jj%G+GU; !Q1((*A!Z/  $65QA Kr:ct\rSrSr\SSSjj5r\SSj5rSrg) TemplatedRingAttentioniBNc U RRn U RRnU RRnUS-U-nS=nnXlXlUR UlUR UlXl [R"UR5UR5/5R5n[R"USU R!5S9nUR#U5n[%U5GHKnUS:aEUUnUR'5nUSUR)U5nUUSR)U5nUS-U-nU "UUUUUUUUUSUS:HU S9 unnU RR*(a>UR-[R.5nUR-[R.5nUR0S:XaUR3S5nUbdU[R4R6R9UU- 5UU- -- nU[R4R6R;UU- 5- nUnUnGMN WR-UR<5nWR?S5nU (aUU4$U$)Nrr gather_dimrTrrr)r) r _ring_mesh_ring_local_rank ring_degree forward_op backward_oprq_shapekv_shaperrcatrrr all_gather_tensor get_groupchunkrangenumel reshape_asconvert_to_fp32rfloat32r,r-rzr{sigmoid logsigmoidrsqueeze)r~rrrrrrrrrrarbr ring_meshr<r6 next_rankprev_outprev_lse kv_bufferikv key_numelrbrcs r6r'TemplatedRingAttention.forwardCsI %<<GG 77HH%==II AX+ ""8#%kk yy /IIs{{}emmo>?JJL ,,Y1IL_L_Lab OOJ/ z"A1uy)IIK )n//49:11%8&]j8 !q&!1 HC 77GGffU]]+ffU]]+xx1}mmB'#!4!4!# UHoRTR5v M! g7fr2)rr)rrrsrs r6rt2TemplatedRingAttention.backward..s!+mJlQDD,@,@Jls'*)rrr^r_r`rrirlrrmrrrcrrdrrerrr rfrgrhrjrkrbrpermute_tensor)r~rrrqr<r6rr next_ranks accum_dtyperrr next_grad_kvrrrrdrurvrwrx grad_query_op grad_key_op grad_value_opgrad_kv_buffergrad_key_numels ` r6r,TemplatedRingAttention.backwards ((@@KK ##;;LL))AAMM AX+ %:./1#5 '*';';'S'S'c'cemmiqiwiw [[KX ;;s||;xW[[[Y  # 1 1EAIIs{{}emmo>?JJL ,,Y1IL_L_Lab OOJ/ z"A1uy)IIK )n//49:11%8&]j8 :EEhO+NO<GG S - 'J  #H - 'J:>!!&H,<,<,> @R@R@T+U!V!a!a!c%44^ZWjWjWlm -#0,n:akJl+m( jZtT4tUY[_aeeer:r3r2r~#torch.autograd.function.FunctionCtxrrrrrrrtorch.Tensor | Nonerfloatrrr float | Nonerrrrr'ParallelConfig' | Noner~rrrrdrerfrgr.r'r,rr3r:r6rYrYBs59J1 0J1J1J1 J1 ' J1  J1J1J1J1J12J1J1X0f 00f0f0fr:rYct\rSrSr\SSSjj5r\SSj5rSrg) TemplatedUlyssesAttentioniNc ^U RRn U RRnU R5mXlXlXlURunnnnURunn nUU-nURUUUUU5RSSSSS5R5nURUUUUU5RSSSSS5R5nURUUUUU5RSSSSS5R5nU4SjXU45upnSXU45upnU "UUUUUUUUUU SU S 9 nU (aUtnnnURXUUU5RSSSSS5R5n[UT5nURSS5RSSSS5R5nU (aqWRXUU5RSSSS5R5n[UT5nURSS5RSSS5R5nOSnU (aUU4$U$) Nr rrr)r*c3<># UHn[UT5v M g7fr2rrrrsrs r6rt4TemplatedUlyssesAttention.forward..sWCVa/599CVc3# UH6oRSS5RSSSS5R5v M8 g7frrr r)Nrr`rrqs r6rtrs8kWjRSYYq!_44Q1a@KKMMWj>ATr]) r _ulysses_meshulysses_degreergrarbrrrr`rrr)r~rrrrrrrrrrarbr ulysses_meshr6rL S_Q_LOCALr5rNrd S_KV_LOCALrOrbrcrs @r6r'!TemplatedUlyssesAttention.forwards\ (??MM %==LL &&(#%/"[[9a!ii:q!z/ aJCKKAqRSUVXYZeegkk!ZWa@HHAqRSUVWbbd aZ!DLLQPQSTVWYZ[ffhWEPUCVWEkX]diWjkE          -   LCqkk!GQ?GG1aQRTUVaac e,kk!Q''1a3>>@ ++aY@HHAqRST__aC$S%0C++a#++Aq!4??ACC'Sz0S0r:c j^ ^ ^^^^URRRnURRRmUR 5mUR um mnm UT-mUR T TTTT 5RSSSSS5R5n[UT5nURSS5RSSSS5R5nURX5tpVpxU U UUU4SjXVU45upn U4SjXU 45upn SXU 45upn XU SSSSSSSSS4 $) Nr rrr)r*c3># UH;nURTTTTT5RSSSSS5R5v M= g7f)rr)rr r*N)rr`r)rrrsrLrNrOrMr6s r6rt5TemplatedUlyssesAttention.backward..sH, @ IIaWgq 9 A A!Q1a P [ [ ] ]@sAAc3<># UHn[UT5v M g7fr2rrs r6rtrs+uRtQ,>q%,H,HRtrc3# UH6oRSS5RSSSS5R5v M8 g7frrrqs r6rtrs9, FhIIaO # #Aq!Q / : : < >7Aqz/##Aw GQGOOPQSTVWYZ\]^iik%h6##Aq)11!Q1=HHJ8;8V5 M, , #-@, ( j,vS]isRt+u( j, GQ]gFh, ( jZtT4tUY[_aeeer:r3r2rrrr3r:r6rrs59<1 0<1<1<1 <1 ' <1  <1<1<1<1<12<1<1|f 0fffr:rct\rSrSr\SSSjj5r\SSj5rSrg) TemplatedRingAnythingAttentioni#Nc ^ Ub [S5eU RRn U R5nU RRnU RR nUS-U-nS=nnXlXlURUl URUl Xl URSn[UU5n[U5m S U 4SjjnU"U5nU"U5n[R"UR!5UR!5/5R#5n[$R&"USUS9nUR)U5nUR+5n[-U5GHjnUS:aMUUnUUnUSUR/U5SS2SU24nUUSR/U5SS2SU24nUS-U-nOUSS2SU24nUSS2SU24nU "UUUUUUUUUSUS:HU S9 unnU RR0(a>UR3[R45nUR3[R45n[7SS 5(aUR9S 5nUbdU[R:R<R?UU- 5UU- -- nU[R:R<RAUU- 5- nUnUnGMm WR3URB5nWRES 5nU (aUU4$U$) NzTemplatedRingAnythingAttention does not support non-None attn_mask: non-uniform sequence lengths across ranks make cross-rank mask slicing ambiguous.rc>TURS- nUS:XaU$URSU/URSSQ7n[R"XRU5/SS9$)Nrrr r)rrre new_zeros)tpad_len pad_shapes_maxs r6 pad_to_s_max.pad_to_s_maxLs_aggaj(G!|W;qwwqr{;I99aY!78a@ @r:rr[Tr]Causal attention is not yet supported for templated attention.z1GQA is not yet supported for templated attention.rz@Reaching this branch of code is unexpected. Please report a bug.) rrr`rr ring_anythingrr*rYulysses_anythingrr) rrrrrrrrrrarbrs r6%_templated_context_parallel_attentionr sYZZLMM 00<TXX#4$r2r3)r4r5r6r7rs r6mask_mod(_native_flex_attention..mask_mod s e!CDDr:c>UTXX44-$r2r3)scorer4r5r6r7rs r6 score_mod)_native_flex_attention..score_mod sye)KLLLr:zCAttention mask must be either None, a BlockMask, or a 2D/4D tensor.c3H# UHoRSSSS5v M g7frorprqs r6rt)_native_flex_attention.. rvrw)rrrr block_maskrrrr))rr flex_attention BlockMaskcreate_block_maskr8rr is_tensorr,viewr/r.rrrr`)rrrrrrrrrrrrrrjrdrrrbs ` r6_native_flex_attentionr  se IJ*/++'J9))A1aJy.2J2JKK #55 +ZI[`[g[g   # # >>Q !y~~a'8!Y^^A=NPQRI$$ZIR ??ejj ( E(99*dI5<<J  M^__LU8KLE  ' '  C ++aAq !C Jr:cUR[R:Xa1[R"US[ S55nUR US9nOUR US9nU(a"UR up4URUSSU5nU$)a Convert a 2D attention mask to an additive mask, optionally reshaping to 4D for SDPA. This helper is used by both native SDPA and xformers backends to handle both boolean and additive masks. Args: attn_mask: 2D tensor [batch_size, seq_len_k] - Boolean: True means attend, False means mask out - Additive: 0.0 means attend, -inf means mask out target_dtype: The dtype to convert the mask to (usually query.dtype) reshape_4d: If True, reshape from [batch_size, seq_len_k] to [batch_size, 1, 1, seq_len_k] for broadcasting Returns: Additive mask tensor where 0.0 means attend and -inf means mask out. Shape is [batch_size, seq_len_k] if reshape_4d=False, or [batch_size, 1, 1, seq_len_k] if reshape_4d=True. r-inf)rr)rrrwhererrrr )r target_dtype reshape_4drr1s r6_prepare_additive_attn_maskr sw(%**$KK 3f > LL|L4 LL|L4  ) NN:q!Y? r:c U(a [S5eUbpURS:Xa`URSURS:Xa@URSURS:Xa URS5RS5nU cRSXU45upn[R R RUUUUUUUUS9n U RSSSS5n U $[UUUUUUUUU[[U S9 n U $) NzDNative attention backend does not support setting `return_lse=True`.r rrc3H# UHoRSSSS5v M g7frorprqs r6rt$_native_attention.. "P>!?  kk!Q1%" J4         35-   Jr:c Sn U cU(dSXU45upn[RRR[RRRR 5 [RR RUUUUUUUUS9n SSS5 W RSSSS5n O'[UUUUUUUUU[[U S9 n U(aU upU(aX4$U $!,(df  NV=f)Nc3d# UH&oRSSSS5R5v M( g7fro)r`rrqs r6rt*_native_cudnn_attention..Q s+]I\AYYq!Q2==??I\s.0rxrr rr)r) rrz attention sdpa_kernel SDPBackendCUDNN_ATTENTIONr{r|r`rrr rrrrrrrrrrrcrbs r6_native_cudnn_attentionr> s" C ]%V[I\]E XX   + +EHH,>,>,I,I,Y,Y Z((%%BB###%C C[kk!Q1%3         24-   HC#C:,,=[ Zs 0/C// C=c U(a [S5eSXU45upn[RRR [RRR R 5 [RRRUUUUUUUUS9n SSS5 W RSSSS5n U $!,(df  N$=f)NzNNative efficient attention backend does not support setting `return_lse=True`.c3H# UHoRSSSS5v M g7frorprqs r6rt._native_efficient_attention.. rvrwrxrr rr)) rrrzrrrEFFICIENT_ATTENTIONr{r|r`rs r6_native_efficient_attentionr$s s ijjLU8KLE    ' '(:(:(E(E(Y(Y Zhh!!>>!?   [ ++aAq !C J [ Z 6/C Cc Ub [S5eSn U cU(dSXU45upn[RRR [RRR R 5 [RRRUUUSUUUUS9n SSS5 W RSSSS5n O'[UUUSUUUUU[[U S9 n U(aU upU(aX4$U $!,(df  NV=f) Nrc3H# UHoRSSSS5v M g7frorprqs r6rt*_native_flash_attention.. rrwrxrr rr)r) rrrzrrrFLASH_ATTENTIONr{r|r`rrrrs r6_native_flash_attentionr* s "KLL C PU,>,I,I,Y,Y Z((%%BB##%C C[kk!Q1%3         9;-   HC#C:,,=[ Zs >/C== D c U(a [S5eSXU45upn[RRR [RRR R 5 [RRRUUUUUUUUS9n SSS5 W RSSSS5n U $!,(df  N$=f)NzINative math attention backend does not support setting `return_lse=True`.c3H# UHoRSSSS5v M g7frorprqs r6rt)_native_math_attention.. rvrwrxrr rr)) rrrzrrrMATHr{r|r`rs r6_native_math_attentionr/ s deeLU8KLE    ' '(:(:(E(E(J(J Khh!!>>!?   L ++aAq !C J L Kr%c*U(a [S5eUca[XU5n[UUUURS5USSUc&S[R "UR S5- OUSSSU- SSS 9 SnU$[UUUUUSUSU[[US 9 nU$) Nrr rrrrFrrr) rrr,r/rrrrrr) rrrrrrrrrbs r6_native_npu_attentionr1 s\]]/IF "    JJqM 6;m# %++b/22Io   < J4         02-   Jr:cUb [S5eU(a [S5eSXU45upnU[R"URS5- n[ UUUUS9nUR SSSS 5nU$) Nz.`attn_mask` is not supported for XLA attentionzAXLA attention backend does not support setting `return_lse=True`.c3H# UHoRSSSS5v M g7frorprqs r6rt(_native_xla_attention..6 rvrwr)rPrrrRrr rr))rrrrxla_flash_attentionr`)rrrrrrrrbs r6_native_xla_attentionr6% sIJJ\]]LU8KLE DIIekk"o. .E     C ++aAq !C Jr:cUb [S5eSnUc[UUUSUUUS9n U(aU tpn O'[UUUSSUUSU[[US9 n U(aU upU(aX4$U $N/`attn_mask` is not supported for sage attentionrrrFr)rr&rrr) rrrrrrrrrcrbrds r6_sage_attentionr:B sJKK C!  LCq3         13-   HC#C:,,r:cUb [S5eSn[[RRn UcU "UUUSUUUS9n U(aU tpn O'[ UUUSSUUSU[ [US9 n U(aU upU(aX4$U $r8)rrrHr{rrrr) rrrrrrrrrcr5rbrds r6_sage_attention_hubr<u sJKK C !5!>!> ? I ID!  LCq3         53-   HC#C:,,r:c  U(a [S5eURup n URup n Ub [X8U 5n[XXURS9uupupunn//nn[ U5H8nU UnUR UUSU245 UR UUSU245 M: URSS5n[R"USS9n[R"USS9n[UUUU UUUUUS9 nURSUS45nU$)Nz?Sage varlen backend does not support setting `return_lse=True`.rrrr) rPrrrrr r!rrr) rrr2r'rrirrrrer+r)rrrrrrrrrrrdrrrrr r!rrrrrrrrbs r6_sage_varlen_attentionr> s<Z[["'++J1a))A1a( K  / :5<< ONQ0\2N<  {I : aL Q  ]+,5JYJ/0 ==A&L9!,J99[a0L    !!!! C --J+ ,C Jr: c <Ub [S5e[UUUSUUUS9$Nr9rr)rr'rrrrrrrrs r6#_sage_qk_int8_pv_fp8_cuda_attentionrC s8JKK '   r:c <Ub [S5e[UUUSUUUS9$rA)rr(rBs r6(_sage_qk_int8_pv_fp8_cuda_sm90_attentionrE s8JKK ,   r:c <Ub [S5e[UUUSUUUS9$rA)rr)rBs r6$_sage_qk_int8_pv_fp16_cuda_attentionrHs8JKK (   r:c <Ub [S5e[UUUSUUUS9$rA)rr*rBs r6&_sage_qk_int8_pv_fp16_triton_attentionrJ1s8JKK *   r:c U(a [S5eURuppURuppU(a[R"5nOUbURS:XaUR S5nUS-S-S-n[ R"XU U4URURS9n[X0RS9nUUSS2SS2SS2SU24'[S5USS2SS2SS2US24'USS2SS2SS2SU24nOLURS :wa [S 5eURS :Xa!URXX5RU5nU(ayX-S :wa [S 5eX-nURSUS 45nURSUS 45RS S S US 5nURSUS 45RS S S US 5n[R"XX#XF5nU(aUR!SS5nU$)NzFxformers attention backend does not support setting `return_lse=True`.r rrFr)rr r*zDOnly 2D and 4D attention masks are supported for xformers attention.rzKNumber of heads in query must be divisible by number of heads in key/value.rr))rrxopsLowerTriangularMaskr,r/rrrrrrr.type_asrmemory_efficient_attentionr)rrrrrrrrrrrr num_heads_qrdr num_heads_kvoriginal_seq_lenaligned_seq_len aligned_mask mask_additivenum_heads_per_grouprbs r6_xformers_attentionrXLs abb,1KK)J;%(YY"A<,,.   >>Q  )~~a0  01 4:a?O!;;)_Ekk||L 8 M8ELAq"3#3"33 47rFrrHrrrorprirkr{rlrrcontextmanagerrsrrrrrrrrrrr"r%r'r2r8rBrrerdrrrrrrrrrrrrrrrrrrrrr rFunctionr"r9r=r?rArGrRrWrYrrrrrrrhrrrrjrrmrrrrrnrrqrrrr rrrtrrur$rvr*rwr/rxr1ryr6rzr:r<r|r>r}rCr~rErrHrrJrrXr3r:r6rs## !//  !!##>>$LM32!!  % H -/h4I$Pg4h13(*^/?F]/^/1l6NtUk6l%d,BC*,*,b1EdLa1b.0j5HOi5j +FmmO!#' "&(M[#%B !   $H$(!&*##' (,%O CB N$2  #_ # D ((J]]00N*UY* *4Q* !J(N(3(V.).)b  0 0  0 %%'7/'GI (,,.>/./ ""$4/'OQ %))+;/., !!#32 $ $$&6-''=#IE#L ? #'"%)%)%) 7777 7  7  7#7#7#7777777 '!7@7B&*04( ,( ( (  ( # (  (( ((((.(V", ,",",Z&*045- ,5- 5- 5-  5- # 5-  5-5- 5-5-5-5-.5-t!, ,!,!,X&*040- ,0- 0- 0-  0- # 0-  0-0- 0-0-0-0-.0-l, ,,,P&*04;-$,;- ,;- ;- ;-  ;- # ;-  ;-;- ;-;-;-;-.;-!;-|#, ,#,#,V&*04C-$,C- ,C- C- C-  C- # C-  C-C- C-C-C-C-.C-!C-L), ,),),b&*04U-$, 'U- ,U- U- U-  U- # U-  U-U- U-U-U-U-.U-!U-U- !U-"#U-$%U-&'U-p0, ,0,0,p&*04#- ,#- #- #-  #- # #-  #-#- #-#-#-#-.#-V&*04%- ,%- %- %-  %- # %-  %-%- %-%-%-%-.%-PV ,VV0&*04# ,# # #  # # #  ## ####.#N\ ,\\   FYR.U^^,,.2" .- B$$NfU^^44fD\f 7 7\f~ntU^^%<%<ntbIw(?(?Iwp15< < < < <  <  < <<<.<<<F&*g]15g] g] g] g]# g]  g]  g] g]g]g].g]Z## =|L"$&*#+040- 0- 0- 0-# 0-  0-  0- 0-!0-0-.0-0-  0-f##"" =|L"$&*#+041- 1- 1- 1-# 1-  1-  1- 1-!1-1-.1-1-  1-h##)) =|L#$&*#+043 3 3 3# 3  3  33!33.33  3l##%% =|L$&*#+042 2 2 2# 2  2  22!22.22 2j##!! =|L$&*04- - - -# -  -  --.-- -.##%% =|L"$&*#+#04J J J J# J  J  J!JJJJ.JJ  JZ##,, =|L#$&*04.- .- .- .-# .-  .-  .-.-..-.-  .-b##$$ =|L#$&*04   #     .  4##(( =|L$&*043- 3- 3- 3-# 3-  3-  3-3-.3-3- 3-l###%BLQ$&*04&- &- &- &-# &-  &-  &- &-&-.&-&- &-R##+]LI$CG048 8 8 8@ 8  8  888.88 8xLP""+6"DH""J## -"$&*048 8 8 8# 8  8  8 888.88  8v##&& =|L"$&*04-- -- -- --# --  --  -- ------.----  --`##** -$&*04   #      . <##&& =|L"$&*040- 0- 0- 0-# 0-  0-  0- 0-0-0-.0-0-  0-f##%% -$&*04   #      . <##$$ =|L"$&*04- - - -# -  -  --.--  -b##$$ -$&*04   #     . 2###%BLQ"$&*04+- +- +- +-# +-  +-  +-+-.+-+-  +-\##!!#%BLQ"$&*04,- ,- ,- ,-# ,-  ,-  ,-,-.,-,-  ,-^##$$#%BLQ$&*040 0 0 0# 0  0  00.00 0f##220A6 E$&*04   #     . .##770A6 E$&*04   #     . .##330A6 E$&*04   #     . .##550A6 E$&*04   #     . .##!!+]LI$&*04@ @ @ @# @  @  @ @@@.@@ @}o , /+ GsJuvw#!%'+$&*#+( , /(8;^_` % #' ( , /%1!4WXY# $%" , /9!<_`a"'+$,0)(,%*.'0 , /:1#=`ab" , /$5aS8[\]!#$ , /#5aS8[\]!"# , /4QC7Z[\!&sx7 x=y/!z {| .|9?}) x:x55x:= y,y''y,/ z8zz {(#{{ |"|| |6|11|69 }&}!!}&) ~2~~