3j,SSKJrJrJr SSKrSSKJr SSKJr SSK J r SSK J r SSK Jr /S Qr\R R"\ SS \R$S \\R$4S jj55r\R R"\ SS \S\R$S \\R$S\S\\R$4 Sjj55r"SS\R.5r"SS\R.5rg))FinalOptionalTypeN)nn) functional)register_notrace_function)use_fused_attn)apply_rot_embed_cat) Attention AttentionRopemaybe_add_maskresolve_self_attn_maskscores attn_maskcUcU$X-$N)rrs O/home/wildlama/miniconda3/lib/python3.13/site-packages/timm/layers/attention.pyrrs&6>F,>>seq_lenattn is_causalreturnc4U(a-URX4[S55RS5nU$UcSnU$UR[R :Xa<[R "X!RS9nURU)[S55 U$UnU$)Nz-infr)dtype)new_fullfloattriu_rtorchbool zeros_like masked_fill_)rrrr attn_biass rrrsMM7"4eFmDJJ1M      EJJ &$$YjjA  z5=9  rc^\rSrSr%Sr\\\S'SS\S\S\ \S\ \S \S \S \S \S \ S\ S\ \ \ RSS4U4SjjjrSS\R S\ \R S\S\R 4SjjrSrU=r$)r +aStandard Multi-head Self Attention module with QKV projection. This module implements the standard multi-head attention mechanism used in transformers. It supports both the fused attention implementation (scaled_dot_product_attention) for efficiency when available, and a manual implementation otherwise. The module includes options for QK normalization, attention dropout, and projection dropout. fused_attnNdim num_heads attn_head_dimdim_outqkv_biasqk_norm scale_norm proj_bias attn_drop proj_drop norm_layerrc >[TU]5 XS.nU=(d UnUnUcX-S:XdS5eX-nU(dU(a U cS5eX lXlX/-UlUS-Ul[ 5Ul[R"XRS-4SU0UD6Ul U(a U "U40UD6O[R"5Ul U(a U "U40UD6O[R"5Ul [R"U 5UlU(aU "UR40UD6O[R"5Ul[R"URU4SU0UD6Ul[R"U 5Ulg) aInitialize the Attention module. Args: dim: Input dimension of the token embeddings. num_heads: Number of attention heads. attn_head_dim: Dimension of each attention head. If None, computed as dim // num_heads. dim_out: Output dimension. If None, same as dim. qkv_bias: Whether to use bias in the query, key, value projections. qk_norm: Whether to apply normalization to query and key vectors. scale_norm: Whether to apply normalization to attention output before projection. proj_bias: Whether to use bias in the output projection. attn_drop: Dropout rate applied to the attention weights. proj_drop: Dropout rate applied after the output projection. norm_layer: Normalization layer constructor for QK normalization if enabled. devicerNr$dim should be divisible by num_heads  /.S   ?a' O)O O''H j) i+i i)" !, % (*99S--!"3I(IbI4;j0R0 4;j0R0 I.7AJt}}33r{{} IIdmmWK9KK I.rxrrc <URupEnURU5RXESURUR5R SSSSS5nUR S5upn URU5URU 5pUR(a@[R"XU UUR(aURROSUS9nO]XR-nXR!SS 5-n [#X[X#5n [%X5n U R'S S 9n URU 5n X-nUR!SS5RXEUR(5nUR+U5nUR-U5nUR/U5nU$) Nr9rrr dropout_prr()shaperAreshaper)r=permuteunbindrCrDr'Fscaled_dot_product_attentiontrainingr0pr? transposerrsoftmaxr>rFrGr1) rHrLrrBNCrAqkvrr$s rforwardAttention.forwardlsW ''ahhqk!!!4>>4==IQQRSUVXY[\^_`**Q-a{{1~t{{1~1 ??..a#.2mm$..**# AJJA{{2r**D.q MI!$2D<>$'DA KK1  % %aDMM : IIaL IIaL NN1 r) r>r0r'r=rDrFr)rGr1rCrAr?) NNFFFTrPrPNNNNF)__name__ __module__ __qualname____firstlineno____doc__rr!__annotations__intrrrrModuler<r Tensorrf__static_attributes__ __classcell__rJs@rr r +sd  +/%)"!$"!!485/5/5/$C= 5/ c] 5/  5/5/5/5/5/5/!bii15/ 5/5/t15# ||  -   rr c>^\rSrSr%Sr\R R\\ S'SS\ S\ S\ \ S\S\S \ S \ S \ S \ \ S \ \RS\S\S\S\4U4SjjjrSS\ \R"S\ \R"S\4SjjrSrU=r$)r zA Self Attention module with ROPE support. Includes options for: * QK normalization option * Attention output (scale) normalization * Fused or unfused QKV projection support r'r(r)r+r, qkv_fusednum_prefix_tokensr0r1r*r2r-r.r/ rotate_halfcv>[TU]5 UUS.nU=(d UnU nUcX-S:XdS5eX-nU (dU (a U cS5eX lUUlUU-UlUS-UlX`l[5UlXl U(aA[R"XRS-4SU0UD6Ul S=Ul =UlUlOSUl [R"XR4SU0UD6Ul [R"XR4SU0UD6Ul[R"XR4SU0UD6UlU (a U "U40UD6O[R "5UlU (a U "U40UD6O[R "5Ul[R&"U5UlU (aU "UR40UD6O[R "5Ul[R"URU4SU 0UD6Ul[R&"U5Ulg) a3Initialize the Attention module. Args: dim: Input dimension of the token embeddings num_heads: Number of attention heads dim_out: Output dimension. If None, same as dim. qkv_bias: Whether to add a bias term to the query, key, and value projections qkv_fused: Whether to use fused QKV projection (single linear) or separate projections num_prefix_tokens: Number of reg/cls tokens at the beginning of the sequence that should not have position embeddings applied attn_drop: Dropout rate for attention weights proj_drop: Dropout rate for the output projection attn_head_dim: Dimension of each attention head. If None, computed as dim // num_heads. norm_layer: Normalization layer constructor to use for QK and scale normalization qk_norm: Enable normalization of query (Q) and key (K) vectors with norm_layer scale_norm: Enable normalization (scaling) of attention output with norm_layer proj_bias: Whether to use bias in the output projection rotate_half: Use 'half' ROPE layout instead of default 'interleaved' r4Nrr6r7r8r9r:)r;r<r)r=r>r?ryr r'rzrr@rAq_projk_projv_projrBrCrDrEr0rFrGr1)rHr(r)r+r,rxryr0r1r*r2r-r.r/rzr5rrIr=rJs rr<AttentionRope.__init__sL  /.S   ?a' O)O O''H ) i+i i)"   9, % !2(*& yymma&7MhM"MDH6: :DK :$+ DH))CLXLLDK))CLXLLDK))CLXLLDK4;j0R0 4;j0R0 I.7AJt}}33r{{} IIdmmWK9KK I.rroperrc hURupVnURbaURU5nURXVSURUR5R SSSSS5nUR S5upn OURU5RXVURUR5RSS5n URU5RXVURUR5RSS5n URU5RXVURUR5RSS5n URU 5URU 5pUbURn [USS5n [R "U SS2SS2SU 2SS24[#U SS2SS2U S2SS24X-S 9/SS 9R%U 5n [R "U SS2SS2SU 2SS24[#U SS2SS2U S2SS24X-S 9/SS 9R%U 5n UR&(a@[(R*"XU UUR,(aUR.R0OS US 9nO]XR2-n XRS S5-n[5XnX45n[7X5nUR9SS 9nUR/U5nX-nURSS5RXVUR:5nUR=U5nUR?U5nURAU5nU$)aForward pass for the attention module. Args: x: Input tensor of shape (batch_size, sequence_length, embedding_dim) rope: Rotary position embeddings tensor for position-aware attention attn_mask: Optional attention mask to apply during attention computation is_causal: If True, use causal (autoregressive) masking Returns: Tensor of shape (batch_size, sequence_length, dim_out) Nr9rNrrrOrzF)halfrUrPrQrSrT)!rVrArWr)r=rXrYr|r^r}r~rCrDrygetattrr catr type_asr'rZr[r\r0r]r?rrr_r>rFrGr1)rHrLrrrr`rarbrArcrdrenptrrr$s rrfAttentionRope.forwards$''a 88 ((1+C++aAt~~t}}EMMaQRTUWXZ[\CjjmGA! A&&qT^^T]]KUUVWYZ[A A&&qT^^T]]KUUVWYZ[A A&&qT^^T]]KUUVWYZ[A{{1~t{{1~1  ((C46D 1Q4C4]+-@1aq=AQSW-cdjklttuvwA 1Q4C4]+-@1aq=AQSW-cdjklttuvwA ??..a#.2mm$..**# AJJA{{2r**D.q MI!$2D<>$'DA KK1  % %aDMM : IIaL IIaL NN1 r)r>r0r'r=rDr}rFr)ryrGr1rCr|rArzr?r~)rhNTTrrPrPNNFFTFNN)NNF)rjrkrlrmrnr jitrr!rorprrrrrqr<rrrfrsrtrus@rr r sF %% %)!"%&!!+/*.!$" %#F/F/F/c] F/  F/  F/ #F/F/F/$C=F/RYYF/F/F/F/F/F/V,004# :5<<(:  - :  ::rr rri)typingrrrr rtorch.nnrrZ_fxr configr pos_embed_sincosr __all__fxwraprrrrpr!rrqr r rrrrs(( $*"1 U?5<<?HU\\4J??-1 llELL)  ell (` `FLBIILr