3j*SSKrSSKJr SSKJr \R "\5rSrSr Sr Sq S\ 4S jr \RRS S S 9S \R S\R S\R S\R S\R S\R S\S\S\S\S\S\S\S\ S\R 4Sj5r\R(S5rS\R S\SS4S jrS!rS(S"\R0R2S\R S#\R S$\R S%\R S-S\S\S-S&\R S-S\\R S44S'jjrg))N)logging)sdpa_attention_forward) attn_implementationc&[b[$SSKJn URS5SnUR S5up#nU"X$=(d SU(aSOSSS 9nS H/n[ [ XVS55(aM[S US US 35e Uq[$)agLoad the MSA hub kernel once and verify the expected callables are present. The ``attn_implementation`` string may carry a ``paged|`` prefix and/or an ``@`` pin (e.g. ``kernels-staging/msa@v0``); the build currently lives on the repo's ``v0`` branch. The loaded module is cached in a module-level global so registration happens once, not per call. Nr) get_kernel|@rT)revisionversionallow_all_kernels)sparse_atten_func build_k2q_csrzThe MSA kernel loaded from `z` does not expose a callable `zK`. Make sure you request a compatible build, e.g. `kernels-staging/msa@v0`.) _MSA_KERNEL hub_kernelsrsplit partitioncallablegetattr ImportError)r rrepo_id_revkernelfn_names a/home/wildlama/miniconda3/lib/python3.13/site-packages/transformers/integrations/msa_attention.pyload_and_register_msa_kernelr$ s'!'',R0G'',OG +stPQei jF9677.wi7UV]U^_[[ :K ztransformers_msa::sparse_atten) mutates_argsqkvq2k cu_seqlens_q cu_seqlens_ktopk block_sizetotal_k max_seqlen_q max_seqlen_k qheads_per_kvscalingimplreturncD[U 5n[RRUR5 UR UUUUUU U U S9unnUR UUUUUUUUU U USU S9 nSSS5 UR 5$!,(df  WR 5$=f)a Opaque wrapper around the CuTe-DSL CSR build + block-sparse kernel. Registered as a ``torch.library`` custom op so ``torch.compile(fullgraph=True)`` treats the whole CSR-build + attention as a single opaque node (no graph break) and ``reduce-overhead`` CUDA graphs can capture it. The internal ``build_k2q_csr`` output is data-dependent in shape, but it never escapes this op (only the fixed-shape ``[total_q, Hq, D]`` attention output does), so the fake/meta impl below is exact. The op is functional (no input mutation). )r0r2r1 qhead_per_kvT)r,r-r1r2blk_kvcausal softmax_scaleN)r$torchcudadevicerr contiguous)r(r)r*r+r,r-r.r/r0r1r2r3r4r5msa k2q_row_ptr k2q_q_indices attn_outputs r#_msa_sparse_atten_oprD<s2 't ,C   188 $%(%6%6    %%&&7 & " ]++   %%%%!,  %4  ! ! ##5 % $4  ! ! ##s 5B Bc.[R"U5$N)r< empty_like)r(r)r*r+r,r-r.r/r0r1r2r3r4r5s r#_msa_sparse_atten_fakerHus$   A r%querydropoutcURRS:wd0[RR UR5SS:wa [ S5eUR S[:wa[S[S35eURR[:wa[S[S35eUS :wa [S 5eURRnU[;a[S [S US 35eg)aValidate kernel capability, dropout and configured topk once per attention module. Mirrors the flash-attention integration, which checks capability/dropout at model init rather than on every forward. The check is cached on the module so the hot path never re-runs it. There is no SDPA fallback: a sparse layer either runs the MSA kernel or this raises. Serves both prefill (q_len > 1) and single-token decode (q_len == 1) -- decode is just a varlen call with one query slot, so there is no context-length threshold. r=r zMSA block-sparse attention requires an SM100 / Blackwell CUDA device. Select a different `attn_implementation` on unsupported hardware.rz2MSA block-sparse attention only supports head_dim .z4MSA block-sparse attention only supports block_size zYMSA block-sparse attention does not support attention dropout; set `attention_dropout=0`.z1MSA block-sparse attention only supports topk in z, got `z0`. Set `index_topk_blocks` to a supported value.N)r>typer<r=get_device_capability RuntimeErrorshapeMSA_SUPPORTED_HEAD_DIM ValueErrorindexerr/MSA_SUPPORTED_BLOCK_SIZE topk_blocksMSA_SUPPORTED_TOPK)modulerIrJr.s r#_validate_msa_initrZs ||F"ejj&F&Fu||&TUV&W[]&] P   {{200MNdMeefghh ~~  $<<OPhOiijkll#~tuu >> % %D %%?@R?SSZ[_Z`a< <  &r%c^URuppURSURSpX-nURSm[U4Sj[55nUT:wa@UR/URSSQUT- P7S5n[R "UU/SS9nUmUR SS5RX-X5R5nUR SS5RX-X5R5nUR SS5RX-X5R5n[R"SUS-U -U UR[RS9nUS:Xa|UbyUSS-R[R5RS5n[R "[R"SUR[RS9U/5nO6[R"SUS-U -U UR[RS9nUR[R5nURX-T5RS5RU SS5R5n[!UUUUUUTUX-U U UUUR"R$5nURXX5$)Nrrrc36># UHoT:dM Uv M g7frFr&).0tr.s r# $_sparse_attention..sB"4QT qq"4s  )dimr)r>dtype)rRnextrXnew_fullr<cat transposereshaper?aranger>int32tozeros unsqueezeexpandrDconfig_attn_implementation)rYrIkeyvaluer4 block_indicesr/cache_positionbsz num_q_headsq_lenhead_dim num_kv_headsk_lenr3 padded_topkpadr(r)r*r,valid_kr-r+rCr.s @r#_sparse_attentionr}ss(- %Ce))A, ! %/M   r "DB"4BBKd$$%T}':':3B'?%TtAS%TVXY =#"6B?  1%%ck;ITTVA a##CKHSSUA 1%%ck<JUUWA<<C!Gu#4eAHHTYT_T_`L  axN.!"%)--ekk:BB1Eyy%++a "TV]!^_ ||Aa5'8%X]XcXcd   5;; 'C ++ck4 ( 2 21 5 < <\2r R ] ] _C'         **K   s; AAr%rYrprqattention_maskrrc UcURSS-nUc[XX#U4XVS.UD6$[USS5(d[XU5 SUlUR R n URS5n [XX#XgX5n U S4$) zf TODO: this opens a door to per-layer attn implementation which is something we might want lalter on. Nrg)rJr4_msa_validatedFTrs) rRrrrZrrUr/getr}) rYrIrprqr~rJr4rrkwargsr/rsrCs r#msa_attention_forwardrs++b/T)% 3~ ?F [a  6+U 3 36'2 $**JZZ 01N#F3wWarK  r%)NrNNN)r<utilsrsdpa_attentionr get_logger__name__loggerrXrVrSrstrr$library custom_opTensorintfloatrD register_fakerHrZr}nnModuletuplerr&r%r#rs 2   H %$ c89K5$ ||5$ ||5$ ||5$  5$ ,, 5$ ,, 5$ 5$5$5$5$5$5$5$ 5$ \\5$L5$p##$( ell U t :9BB+/ )-  HHOO  <<    <<  LL4'   T\ <<$&  5<<  r%