3jX$SrSSKrSSKJr SSKJrJr SSKr\R"\ 5r /SQr S\ \ S-S\ \ 4Sjr\"S S 9S \ S\4S j5r"S S\5r\R$R'S0S9S:S\R(S\R(S\R(S\R(S\R(S-S\ S\ S\S\S-S\ \ S-S\S\R(S-S\R(S-S\ S-S\\R(\R(\R(44Sjj5r\R0S:S\R(S\R(S\R(S\R(S\R(S-S\ S\ S\S\S-S\ \ S-S\S\R(S-S\R(S-S\ S-S\\R(\R(\R(44S jj5rSSS!SSSSS".S\R(S\R(S\R(S\R(S\R(S-S\ S\ S#\S-S\S-S\\ \ 4S\S\R(S-S\R(S-S\ S-S\R(\\R(\R(4-4S$jjr\R$R'S%S&1S9S:S&\R(S\R(S\R(S\R(S\R(S\R(S-S\ S\ S\S\S-S\ \ S-S\S\R(S-S\R(S-S\ S-S\R(4 S'jj5r\R0S:S&\R(S\R(S\R(S\R(S\R(S\R(S-S\ S\ S\S\S-S\ \ S-S\S\R(S-S\R(S-S\ S-S\R(4 S(jj5rSSS!SSSSS".S&\R(S\R(S\R(S\R(S\R(S\R(S-S\ S\ S#\S-S\S-S\\ \ 4S\S\R(S-S\R(S-S\ S-S\R(\\R(\R(4-4 S)jjrS*\S+\\S,4S-\SS4S.jr\R$R'S/0S9S;S0\R(S\R(S\R(S\R(S&\R(S1\R(S\R(S\R(S\ S\ S\S2\R(S\S-S\ \ S-S\\R(\R(\R(44S3jj5r\R0S;S0\R(S\R(S\R(S\R(S&\R(S1\R(S\R(S\R(S\ S\ S\S2\R(S\S-S\ \ S-S\\R(\R(\R(44S4jj5r S*\S0\R(S5\R(S6\R(S\\R(S-S,44 S7jr!\RE\!\S89 \RFRI\RJRLRN5 SS9K(J)r)J*r*J+r+J,r, \*\,\RJRZR.'\+\,\RJRZR6'\)\,\RJRZR>'g)R ;q>R#7f   K$;T 88            9  *0F1IvayY Y @A/4yy~~/V/V         #(^)!n#!!0W0 , Y1& ELLJ ; **rc[U 5n [R"U5nURS5nURS5n[R"UU4[R UR S9n[RR(a[RR5nU[RRR:XaEURS5S- n[R"UUU4[R UR S9n[R"S[RUR S9nUUU4$)z Fake implementation for meta tensor computation and tracing. Based on the 3D varlen path from meta__flash_attention_forward: - query shape: (total, num_heads, head_dim) - logsumexp shape: (num_heads, total_q) rr4r>r=)rrF empty_likesizeemptyfloatr@versionhip_C_get_rocm_fa_preferred_backend_ROCmFABackendAOTritonrL)r%r&r'r(r)r*r+r,r-r r.r/r0r1rOtotal_q num_heads logsumexp preferred batch_sizerQs r_varlen_attn_fakeres0)5K  e $FjjmG 1 I GEKK I }}HH;;= //88 8!q)A-J Y.ekk%,,I D U\\JI 9i ''r)r r ) return_auxr-r r.r/r0r1rfcURS5nU bURS5OURS5nU (dX:wa[SUSUS35eU (aX-S:wa[SUSUS35eU S :Hn[RRR UUUUUUUUU[ U 5U U U U 5unnnUbUR(aUU4$U$) a-Compute variable-length attention using Flash Attention. This function is similar to scaled_dot_product_attention but optimized for variable-length sequences using cumulative sequence position tensors. Args: query (Tensor): Query tensor; shape :math:`(T_q, H_q, D)` key (Tensor): Key tensor; shape :math:`(T_k, H_{kv}, D)`, or :math:`(\text{total\_pages}, \text{page\_size}, H_{kv}, D)` when ``block_table`` is provided. value (Tensor): Value tensor; shape :math:`(T_k, H_{kv}, D)`, or :math:`(\text{total\_pages}, \text{page\_size}, H_{kv}, D)` when ``block_table`` is provided. cu_seq_q (Tensor): Cumulative sequence positions for queries; shape :math:`(N+1,)` cu_seq_k (Tensor): Cumulative sequence positions for keys/values; shape :math:`(N+1,)` max_q (int): Maximum query sequence length in the batch. max_k (int): Maximum key/value sequence length in the batch. return_aux (Optional[AuxRequest]): If not None and ``return_aux.lse`` is True, also returns the logsumexp tensor. scale (float, optional): Scaling factor for attention scores window_size (tuple[int, int], optional): Window size for sliding window attention as (left, right). Use (-1, -1) for full attention (default), (-1, 0) for causal attention, or (W, 0) for causal attention with sliding window of size W. enable_gqa (bool): If set to True, enables Grouped Query Attention (GQA) and allows key/value to have fewer heads than query. Each KV head is shared by a group of :math:`H_q / H_{kv}` query heads, so :math:`H_q` must be divisible by :math:`H_{kv}`. Default is False. seqused_k (Tensor, optional): Number of valid KV tokens per batch element; shape :math:`(N,)`. When set, only the first ``seqused_k[i]`` tokens in the key/value sequence for batch element *i* participate in attention. Useful for KV-cache decoding where the cache slot is larger than the actual sequence. Inference-only (not supported in backward). block_table (Tensor, optional): Block table for paged KV cache; shape :math:`(N, \text{max\_pages\_per\_seq})`, dtype ``int32``. Requires ``seqused_k``. Inference-only (not supported in backward). When ``block_table`` is provided, ``key`` and ``value`` are a "pool" of pages of tokens of KV data and the pages belong to any sequence/order. The ``block_table`` is what maps each sequence's logical chunks back to physical pages in this pool. ``seqused_k[i]`` tells the kernel how many tokens in sequence *i* are actually valid, since the last page is typically only partially filled. num_splits (int, optional): Number of splits for split-KV. Set to ``1`` to disable split-KV which enables batch invariance. Split-KV parallelizes the key/value sequence dimension across multiple thread blocks and combines partial results. The split decision depends on ``max_k`` (the longest sequence in the batch), so different batch compositions can change the reduction order and produce different floating-point results for the same sequence. When this is disabled, bitwise identical outputs are guaranteed for a given sequence regardless of what other sequences are in the batch, at the cost of lower GPU utilization when there are few queries. When ``None`` (default), the kernel chooses automatically. Returns: output (Tensor): Output tensor from attention computation; shape :math:`(T_q, H_q, D)`. If ``return_aux`` is not None and ``return_aux.lse`` is True: lse (Tensor): Log-sum-exp of attention scores; shape :math:`(T_q, H_q)`. Shape legend: - :math:`N`: Batch size - :math:`T_q`: Total number of query tokens in the batch (sum of all query sequence lengths) - :math:`T_k`: Total number of key/value tokens in the batch (sum of all key/value sequence lengths) - :math:`H_q`: Number of query attention heads - :math:`H_{kv}`: Number of key/value attention heads (equal to :math:`H_q` unless GQA is enabled) - :math:`D`: Head dimension Example:: >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_CUDA) >>> batch_size, max_seq_len, embed_dim, num_heads = 2, 512, 1024, 16 >>> head_dim = embed_dim // num_heads >>> seq_lengths = [] >>> for _ in range(batch_size): ... length = torch.randint(1, max_seq_len // 64 + 1, (1,)).item() * 64 ... seq_lengths.append(min(length, max_seq_len)) >>> seq_lengths = torch.tensor(seq_lengths, device="cuda") >>> total_tokens = seq_lengths.sum().item() >>> >>> # Create packed query, key, value tensors >>> query = torch.randn( ... total_tokens, num_heads, head_dim, dtype=torch.float16, device="cuda" ... ) >>> key = torch.randn( ... total_tokens, num_heads, head_dim, dtype=torch.float16, device="cuda" ... ) >>> value = torch.randn( ... total_tokens, num_heads, head_dim, dtype=torch.float16, device="cuda" ... ) >>> >>> # Build cumulative sequence tensor >>> cu_seq = torch.zeros(batch_size + 1, device="cuda", dtype=torch.int32) >>> cu_seq[1:] = seq_lengths.cumsum(0) >>> max_len = seq_lengths.max().item() >>> >>> # Call varlen_attn >>> output = varlen_attn( ... query, key, value, cu_seq, cu_seq, max_len, max_len ... ) r4r GExpect query and key/value to have the same number of heads but got Hq= and Hkv=&. Try setting enable_gqa=True for GQA.rMExpect number of query heads to be a multiple of kv heads for GQA but got Hq=.r r)rWrrFrG torch_attnrTlistr)r%r&r'r(r)r*r+rfr-r r.r/r0r1 num_heads_q num_heads_kr,outrrRs rrrsj**Q-K!,!8#((1+chhqkK +4%i }=3 4  k/14%i }A ?  w&I))&&33       [KCa *..Cx Jrztorch_attn::_varlen_attn_outrrcH[U 5n UR=(a [URR5nU(a [ S5e[ RS5 [RRRUUUUUUUUSUSU U SU SU U US9nU$)z Private custom op for variable-length attention with pre-allocated output. Same as _varlen_attn but writes the attention output into the provided out tensor. z+cuDNN backend does not support out variant.z1Using Flash Attention backend for varlen_attn_outr6Frr4)r-r;r<r/r0r1) rrArr@rBrErCrDrFrGrH+_flash_attention_forward_no_dropout_inplace)rrr%r&r'r(r)r*r+r,r-r r.r/r0r1rMrPs r_varlen_attn_outruRs,)5K G"3ELL4F4F"GIHIIHH @A))..LL        $Q%a.#MK( rcURS5nURS5n[R"UU4[RURS9n[R R (a[RR5nU[RRR:XaEURS5S- n[R"UUU4[RURS9nU$)> Fake implementation for meta tensor computation and tracing. rr4r>) rWrFrXrYr@rZr[r\r]r^r_)rrr%r&r'r(r)r*r+r,r-r r.r/r0r1r`rarbrcrds r_varlen_attn_out_fakerxs*jjmG 1 I GEKK I }}HH;;= //88 8!q)A-J Y.ekk%,,I rcURS5nU bURS5OURS5nU (dUU:wa[SUSUS35eU (aUU-S:wa[SUSUS35eU S :Hn[RRR UUUUUUUUUU [ U 5U U U U5nUbUR(aUU4$U$) zCompute variable-length attention using Flash Attention with a pre-allocated output tensor. Same as :func:`varlen_attn` but writes the attention output into the provided ``out`` tensor instead of allocating a new one. r4r rhrirjrrkrlrm)rWrrFrGrnruror)rrr%r&r'r(r)r*r+rfr-r r.r/r0r1rprqr,rs rrrs0**Q-K!,!8#((1+chhqkK +4%i }=3 4  kK/14%i }A ?  w&I ))   / /        [ C"*..Cx Jrctxinputs.rOcUunnnnnnn n n n n nnnUunnnUb [S5eUb [S5eURX4XVUUUU5 XlXlXlXlXlg)Nz)seqused_k is an inference-only parameter.z+block_table is an inference-only parameter.)rEsave_for_backwardr*r+r,r-r )rzr{rOr%r&r'r(r)r*r+r,r-r r.r/r0r1rrrrQs r_setup_contextr~s        CiFGGHII%exc9UIIMI!Orz!torch_attn::_varlen_attn_backwardgrad_outrrQcV[U 5n [R"SURS9nUR=(a [ URR 5nU(al[RS5 U SS:wd U SS:wa [S5e[RRRUUUUUUUUUU SU U UU S9unnnOV[RS 5 [RRRUUUUUUUUUU SU U UU U SU SS 9unnnUUU4$) Nr)r@r3r r4r5r6r7r9)r-r;r<)rrFrXr@rArrBrCrDrErGrH_cudnn_attention_backward_flash_attention_backward)rr%r&r'rrrr(r)r*r+r,rQr-r unusedrMdqdkdvs r_varlen_attn_backwardr sD")5K [[5<< 0F G"3ELL4F4F"GI 67 q>R ;q>R#7f YY^^==              > B$ @AYY^^==              (^)!n#> B& r2:rc[U 5n [R"U5n[R"U5n[R"U5nXU4$)rw)rrFrV)rr%r&r'rrrr(r)r*r+r,rQr-r grad_querygrad_key grad_values r_varlen_attn_backward_fakerQsI()5K!!%(J$H!!%(J  ++rgrad_lsegrad_rngc*URupEpgppURn URn URnURnUR n[ RRRUUUUU U UUU U UU UU5unnnSnUUU/SU-Q7$)N )N) saved_tensorsr*r+r,r-r rFrGrnr)rzrrrr%r&r'r(r)rrrrQr*r+r,r-r rrr num_paramss r _backwardrnsBEARAR>E IIE IIE I IIE//K%%;;        JBB$J B 0'J. 00r) setup_context)_varlen_attn_backward_flop_varlen_attn_forward_flop_varlen_attn_out_flop flop_registry)FNNFNNN)NN).r logging functoolsrtypingrrrF getLoggerrrC__all__rointrr!rrlibrary custom_opTensorrYtuplerT register_fakererrurxrr~rrrregister_autograd_dynamodisallow_in_graphrGrHrttorch.utils.flop_counterrrrrrnrrrrsa " ! :S D(8T#Y 1CD 3"E$(%)'+!V+ <<V+ V+ <<V+ll V+ llT! V+  V+ V+V+ 4<V+cT!V+V+||d"V+$V+d V+ 5<<u|| 34V+FV+r$(%)'+!.( <<.( .( <<.(ll .( llT! .(  .( .(.( 4<.(cT!.(.(||d".($.(d .( 5<<u|| 34.(.(t%)#+%)'+!V <<V V <<Vll V llT! V  V VT!V 4<VsCxVV||d"V$Vd V  \\E%,, 455!Vr7ugN$(%)'+!2 2 <<2 2 << 2 ll 2 llT! 2 2 22 4<2cT!22||d"2$2d 2  \\!2O2j$(%)'+!" " <<" " << " ll " llT! " " "" 4<"cT!""||d""$"d "  \\!" "^%)#+%)'+!!: : <<: : << : ll : llT! : : :T!: 4<:sCx::||d":$: d !:" \\E%,, 455#:z""U38_"c"d"B<2N$(AllA <<A A << A  A  AllAllA A AA||A 4<AcT!A 5<<u|| 34AOAH$$$(,ll, <<, , << ,  ,  ,ll,ll, , ,,||, 4<,cT!, 5<<u|| 34,%,81 1 105 1HM 1 5<<$  #$1B yG  IINN>>4M eii""//07L eii""334