3jR &SrSSKrSSKrSSKJr SSKJrJrJrJ r J r J r J r J r SSKrSSKJr SSKJs Jr SSKJrJrJrJr SSKJrJrJrJrJrJrJ r J!r!J"r"J#r#J$r$J%r%J&r&J'r'J(r(J)r)J*r*J+r+J,r,J-r-J.r. SSK/J0r0 SS K1J2r2 SS K3J4r4 SS K5J6r6J7r7 S /r8"S S\Rr5r:"SS\Rr5r;"SS\Rr5r<"SS \Rr5r=SS\\>\R~4S\RrS\>S\\>\R~44Sjjr@SS\\>\R~4S\RrS\>S\AS\\>\R~44 SjjrBSS\>S\AS\=4SjjrCSS\>S\\>\44S jjrDSS\>S\\>\44S!jjrESS\>S\\>\44S"jjrF\6"0S#\D"S$S%9_S&\D"S$S'S(S)S*9_S+\D"S$\\S'S(S)S,9_S-\D"S$\\S.S(S)S,9_S/\D"S$S0S(S)S*9_S1\D"S$S0S(S)S*9_S2\D"S$S0S(S)S*9_S3\D"S$S'S(S49_S5\D"S$S'S(S49_S6\D"S$S0S(S49_S7\D"S$S0S(S49_S8\D"S$S0S(S49_S9\D"S$S0S(S)S:S;9_S<\D"S$S0S(S)S:S;9_S=\D"S$S0S(S)S:S;9_S>\D"S$SS?9_S@\D"S$SS?9_0SA\D"S$SS?9_SB\D"S$SS?9_SC\D"S$SS?9_SD\D"S$SES?9_SF\D"S$SES?9_SG\D"S$SHS?9_SI\D"S$SJS?9_SK\D"S$S'S(SJSL9_SM\D"S$SES?9_SN\D"S$SES?9_SO\D"SSP9_SQ\D"S$SRSSSTSTSU9_SV\D"S$SRSSSTSTSU9_SW\D"S$SRSSS49_SX\D"S$SRSSSTSTSU9_SY\E"S$SZSHS[9_S\\E"S$SZSHS[9_E0S]\E"S$S^SES[9_S_\E"S$S'SES[9_S`\E"S$S0SaS[9_Sb\E"S$S0SS[9_Sc\E"S$S0SS[9_Sd\E"S$S0SS[9_Se\E"S$S0SS[9_Sf\E"S$SgSS[9_Sh\E"S$SgSS[9_Si\E"S$SgSS[9_Sj\E"S$S0SS[9_Sk\E"S$S0SS[9_Sl\D"S$\\SmSn9_So\D"S$\\SmSn9_Sp\D"S$\\SmSn9_Sq\D"S$\\SmSn9_Sr\D"S$\\SmSn9_E0Ss\D"S$\\SmSn9_St\D"S$\\SmSn9_Su\D"S$\\SmSn9_Sv\D"S$\\SmSn9_Sw\D"S$\\SmSn9_Sx\D"S$\\SmSn9_Sy\D"S$\\SmSn9_Sz\F"S$S%9_S{\F"S$S%9_S|\F"S$S%9_S}\F"S$S%9_S~\F"S$S%9_S\F"S$S%9_S\F"S$S%9_S\F"S$S%9_S\F"S$SSS9_S\F"S$SSS9_E\F"S$S%9\F"S$S%9\F"S$S%9\F"S$SSS9S.E5rG\7SS\AS\=4Sjj5rH\7SS\AS\=4Sjj5rI\7SS\AS\=4Sjj5rJ\7SS\AS\=4Sjj5rK\7SS\AS\=4Sjj5rL\7SS\AS\=4Sjj5rM\7SS\AS\=4Sjj5rN\7SS\AS\=4Sjj5rO\7SS\AS\=4Sjj5rP\7SS\AS\=4Sjj5rQ\7SS\AS\=4Sjj5rR\7SS\AS\=4Sjj5rS\7SS\AS\=4Sjj5rT\7SS\AS\=4Sjj5rU\7SS\AS\=4Sjj5rV\7SS\AS\=4Sjj5rW\7SS\AS\=4Sjj5rX\7SS\AS\=4Sjj5rY\7SS\AS\=4Sjj5rZ\7SS\AS\=4Sjj5r[\7SS\AS\=4Sjj5r\\7SS\AS\=4Sjj5r]\7SS\AS\=4Sjj5r^\7SS\AS\=4Sjj5r_\7SS\AS\=4Sjj5r`\7SS\AS\=4Sjj5ra\7SS\AS\=4Sjj5rb\7SS\AS\=4Sjj5rc\7SS\AS\=4Sjj5rd\7SS\AS\=4Sjj5re\7SS\AS\=4Sjj5rf\7SS\AS\=4Sjj5rg\7SS\AS\=4Sjj5rh\7SS\AS\=4Sjj5ri\7SS\AS\=4Sjj5rj\7SS\AS\=4Sjj5rk\7SS\AS\=4Sjj5rl\7SS\AS\=4Sjj5rm\7SS\AS\=4Sjj5rn\7SS\AS\=4Sjj5ro\7SS\AS\=4Sjj5rp\7SS\AS\=4Sjj5rq\7SS\AS\=4Sjj5rr\7SS\AS\=4Sjj5rs\7SS\AS\=4Sjj5rt\7SS\AS\=4Sjj5ru\7SS\AS\=4Sjj5rv\7SS\AS\=4Sjj5rw\7SS\AS\=4Sjj5rx\7SS\AS\=4Sjj5ry\7SS\AS\=4Sjj5rz\7SS\AS\=4Sjj5r{\7SS\AS\=4Sjj5r|\7SS\AS\=4Sjj5r}\7SS\AS\=4Sjj5r~g)aJ EVA EVA ViT from https://github.com/baaivision/EVA , paper: https://arxiv.org/abs/2211.07636 This file contains a number of ViT variants the utilise ROPE position embeddings, SwiGLU and other additions: * EVA & EVA02 model implementations that evolved from BEiT, additional models in vision_transformer.py. * `timm` original SBB ViT w/ ROPE position embeddings * Perception Encoder (PE) ViT from Meta (https://arxiv.org/abs/2504.13181) * ROPE-ViT from Naver AI (https://arxiv.org/abs/2403.13298) * DINOv3 from META AI Research (https://arxiv.org/abs/2508.10104) @article{EVA, title={EVA: Exploring the Limits of Masked Visual Representation Learning at Scale}, author={Fang, Yuxin and Wang, Wen and Xie, Binhui and Sun, Quan and Wu, Ledell and Wang, Xinggang and Huang, Tiejun and Wang, Xinlong and Cao, Yue}, journal={arXiv preprint arXiv:2211.07636}, year={2022} } EVA-02: A Visual Representation for Neon Genesis - https://arxiv.org/abs/2303.11331 @article{EVA02, title={EVA-02: A Visual Representation for Neon Genesis}, author={Fang, Yuxin and Sun, Quan and Wang, Xinggang and Huang, Tiejun and Wang, Xinlong and Cao, Yue}, journal={arXiv preprint arXiv:2303.11331}, year={2023} } @article{bolya2025perception, title={Perception encoder: The best visual embeddings are not at the output of the network}, author={Bolya, Daniel and Huang, Po-Yao and Sun, Peize and Cho, Jang Hyun and Madotto, Andrea and Wei, Chen and Ma, Tengyu and Zhi, Jiale and Rajasegaran, Jathushan and Rasheed, Hanoona and others}, journal={arXiv preprint arXiv:2504.13181}, year={2025} } @inproceedings{heo2024rotary, title={Rotary position embedding for vision transformer}, author={Heo, Byeongho and Park, Song and Han, Dongyoon and Yun, Sangdoo}, booktitle={European Conference on Computer Vision}, pages={289--305}, year={2024}, organization={Springer} } @article{simeoni2025dinov3, title={{DINOv3}}, author={Sim{'e}oni, Oriane and Vo, Huy V. and Seitzer, Maximilian and Baldassarre, Federico and Oquab, Maxime and Jose, Cijo and Khalidov, Vasil and Szafraniec, Marc and Yi, Seungeun and Ramamonjisoa, Micha{"e}l and Massa, Francisco and Haziza, Daniel and Wehrstedt, Luca and Wang, Jianyuan and Darcet, Timoth{'e}e and Moutakanni, Th{'e}o and Sentana, Leonel and Roberts, Claire and Vedaldi, Andrea and Tolan, Jamie and Brandt, John and Couprie, Camille and Mairal, Julien and J{'e}gou, Herv{'e} and Labatut, Patrick and Bojanowski, Piotr}, year={2025}, eprint={2508.10104}, url={https://arxiv.org/abs/2508.10104}, } DINOv3 code was a modification of existing EVA model and support modules, so licensed under Apache-2.0 like timm. Weights from META remain under DINOv3 License (https://ai.meta.com/resources/models-and-libraries/dinov3-license/). Modifications by / Copyright 2023 Ross Wightman, original copyrights below N)partial)AnyCallableDictListOptionalSetTupleUnion)IMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STDOPENAI_CLIP_MEANOPENAI_CLIP_STD) PatchEmbedMlpGluMlpSwiGLU LayerNormDropPathcalculate_drop_path_ratesPatchDropoutWithIndicescreate_rope_embedapply_rot_embed_catapply_keep_indices_nlc trunc_normal_resample_patch_embedresample_abs_pos_embedglobal_pool_nlc to_2tupleuse_fused_attnmaybe_add_maskresolve_self_attn_mask AttentionRopeAttentionPoolLatent)build_model_with_cfg)feature_take_indices) checkpoint)generate_default_cfgsregister_modelEvac<^\rSrSr%Sr\R R\\ S'SS\ S\ S\S\S\S \ S \ S \ S \ \ S \ \ S\S\S\4U4SjjjrSSjrSSjrSS\ \R"S\ \R"S\4SjjrSSjrSrU=r$) EvaAttentionizFEVA Attention with ROPE, no k-bias, and fused/unfused qkv options fused_attndim num_headsqkv_bias qkv_fusedqkv_bias_separatenum_prefix_tokens attn_drop proj_drop attn_head_dim norm_layerqk_norm scale_norm rotate_halfc>XS.n[TU]5 U (dU (a U cS5eX lX-UlU bXlURUR-nURS-UlX`l[ 5UlXPlXl U(a[R"UUS-4SS0UD6Ul S=Ul =UlUlU(a[R "["R$"U40UD65UlUR)S["R$"U40UD6SS 9 [R "["R$"U40UD65UlOS=Ul=UlUlOy[R"UU4SU0UD6Ul [R"UU4SS0UD6Ul[R"UU4SU0UD6UlSUl S=Ul=UlUlU (aU "UR40UD6O[R."5UlU (aU "UR40UD6O[R."5Ul[R4"U5UlU (a U "U40UD6O[R."5Ul[R"UU40UD6Ul[R4"U5UlUR?5 g) a Args: dim: Input dimension of the token embeddings num_heads: Number of attention heads qkv_bias: Whether to add a bias term to the query, key, and value projections qkv_fused: Whether qkv projections are fused into one projection or separate qkv_bias_separate: Whether to apply bias to qkv as a separate addition or part of F.linear() call num_prefix_tokens: Number of reg/cls tokens at the beginning of the sequence that should not have position embeddings applied attn_drop: Dropout rate for attention weights proj_drop: Dropout rate for the output projection attn_head_dim: Dimension of each attention head (if None, computed as dim // num_heads) norm_layer: Normalization layer constructor to use for QK and scale normalization qk_norm: Enable normalization of query (Q) and key (K) vectors with norm_layer scale_norm: Enable normalization (scaling) of attention output with norm_layer rotate_half: Use half rotation layout instead of interleaved devicedtypeNz>1]]d* !2(*!2& yyhlEE"EDH6: :DK :$+  ll5;;x+F2+FG $$Xu{{8/Jr/JW\$] ll5;;x+F2+FG :>> >dkDK))CGGBGDK))CDDDDK))CGGBGDKDH6: :DK :$+ 9@j5"5bkkm 9@j5"5bkkm I.2>$'DA KK1  % %aA . IIaL IIaL NN1 rbc$UR5 g)z"Initialize non-persistent buffers.N)rfrgs r`init_non_persistent_buffers(EvaAttention.init_non_persistent_bufferss rb)r6r/rGrCrWrMrYr1r5rZr7rRrVrLrKr4r<rHrTrN)TTFr%rvrvNNFTFNNreturnNNNF)__name__ __module__ __qualname____firstlineno____doc__rPjitFinalbool__annotations__intfloatrrrFr[rfTensorrr__static_attributes__ __classcell__r_s@r`r-r-isC %% !"&+%&!!+/-1!# %!I I I  I  I $ I  #I I I $C=I !*I I I I I V ,004# A5<<(A  - A  AFrbr-c(P^\rSrSrSSSSSSSSSSSSSS \R \S S S 4S \S \S \S \S\ S\S\S\S\S\S\ S\S\ S\ S\ S\ \ S\ S\ S\ \4&U4Sjjjr S&SjrS'S \R S!\ \R S"\ \R S#\S\R 4 S$jjrS%rU=r$)(EvaBlocki T@Frr%evarvNr0r1r2r3 mlp_ratio swiglu_mlpswiglu_align_to scale_mlpscale_attn_innerr5 attn_typer<r7r6 drop_path init_values act_layerr9r8c >UUS.n[TU]5 U"U40UD6UlU S:Xa[O[nU"U4UUUU UU UUU U S. UD6UlUUlUb+[R"[R"U40UD65OSUl US:a [U5O[R"5UlU"U40UD6Ul[!X-5nU(a`U(dU(a [#S UUU(aUOSU US.UD6UlOQ['S UUS-U(aUOS[R(SU S .UD6UlO[+S UUUU(aUOSU S .UD6UlUb+[R"[R"U40UD65OSUlUS:a [U5O[R"5UlUR15 g) a>Initialize the EVA transformer block. Args: dim: Input dimension of the token embeddings num_heads: Number of attention heads qkv_bias: Whether to use bias terms in query, key, value projections qkv_fused: Whether to use a single projection for query, key, value mlp_ratio: Ratio of MLP hidden dimension to input dimension swiglu_mlp: Whether to use SwiGLU activation in the MLP scale_mlp: Whether to use normalization in the MLP scale_attn_inner: Whether to use normalization within the attention mechanism num_prefix_tokens: Number of tokens at the beginning of the sequence (class tokens, etc.) attn_type: Type of attention module to use ('eva' or 'rope') proj_drop: Dropout rate for projection layers attn_drop: Dropout rate for attention matrix drop_path: Stochastic depth rate init_values: Initial value for LayerScale, None = no LayerScale act_layer: Activation layer constructor norm_layer: Normalization layer constructor attn_head_dim: Dimension of each attention head (if None, computed as dim // num_heads) r>rl r1r2r3r5r6r7r8r9r;r<Nrv in_featureshidden_featuresr9dropalign_torrFrrr9r gate_lastrrrrr9r)rErFnorm1r#r-rrrIrOrPrQgamma_1rrU drop_path1norm2rrmlprSiLUrgamma_2 drop_path2r[)r\r0r1r2r3rrrrrr5rr<r7r6rrrr9r8r?r@kwargsr]attn_clsrr_s r`rFEvaBlock.__init__s\ / *r* $-$7=\   /'!'#     '?J?Vr||EKK$:r$:;\` 1:R(9-R[[]*r* co. O! #$3-6zD",   " #$3a$7-6zD gg#"  /#)2:   DH@K?Vr||EKK$:r$:;\` 1:R(9-R[[] rbrcURbi[RRURUR5 [RRUR UR5 gg)zInitialize parameters.N)rrIrd constant_rrrgs r`r[EvaBlock.reset_parameters{sM << # GG  dllD,<,< = GG  dllD,<,< = $rbrrlrmrnc URcdXRURURU5X#US95-nXR UR UR U555-nU$XRURURURU5X#US9-5-nXR URUR UR U55-5-nU$Nrlrmrn)rrrrrrrrr\rrlrmrns r`rEvaBlock.forwards << OODIIdjjm$gpI$qrrAOODHHTZZ]$;<,004# || 5<<(   -      rbrc(^^\rSrSrSrSSSSSSSSSSS S S S \R \RS S S 4S \S \S \ S\ S\ S\ S\ S\ S\S\ S\ S\S\ S\ S\ S\ \ S\ S\ S\ \4&U4SjjjrS&S\R S \ \R S!\ \R S"\ S#\R 4 S$jjrS%rU=r$)'EvaBlockPostNormizEEVA block w/ post-norm and support for swiglu, MLP norm scale, ROPE. TrrFrr%rvNr0r1r2r3rrr<rrrrr5r7r6rrrr9r8c >UUS.n[TU]5 US:Xa[O[nU"U4UUUU UU UUU US. UD6UlU"U40UD6UlUS:a [ U5O[R"5Ul [X-5nU(aYU (a [S UUU (aUOSU U S.UD6Ul OQ[S UUS-U (aUOS[RSU S .UD6Ul O[S UUUU (aUOSU S .UD6Ul U"U40UD6UlUS:a[ U5Ulg[R"5Ulg) alInitialize the post-norm EVA transformer block. Args: dim: Input dimension of the token embeddings num_heads: Number of attention heads qkv_bias: Whether to use bias terms in query, key, value projections qkv_fused: Whether to use a single projection for query, key, value mlp_ratio: Ratio of MLP hidden dimension to input dimension swiglu_mlp: Whether to use SwiGLU activation in the MLP scale_mlp: Whether to use normalization in the MLP scale_attn_inner: Whether to use normalization within the attention mechanism num_prefix_tokens: Number of tokens at the beginning of the sequence (class tokens, etc.) attn_type: Type of attention module to use ('eva' or 'rope') proj_drop: Dropout rate for projection layers attn_drop: Dropout rate for attention matrix drop_path: Stochastic depth rate init_values: Initial value for LayerScale, None = no LayerScale (NOTE: ignored for post-norm block) act_layer: Activation layer constructor norm_layer: Normalization layer constructor attn_head_dim: Dimension of each attention head (if None, computed as dim // num_heads) r>rlrrvNrrrFrrr)rErFr#r-rrrrIrUrrrrrrrrr)r\r0r1r2r3rrr<rrrrr5r7r6rrrr9r8r?r@rr]rrr_s r`rFEvaBlockPostNorm.__init__so\ / $-$7=\   /'!'#      *r* 1:R(9-R[[]co. ! #$3-6zD",   " #$3a$7-6zD gg#"  /#)2:   DH *r* 1:R(9-R[[]rbrrlrmrnrc XRURURXX4S955-nXRUR UR U555-nU$r)rrrrrrrs r`rEvaBlockPostNorm.forwardsR  499QY9+l mn n  488A; 78 8rb)rrrrrrr)rrrrrrIrrrrrrrrrFrPrrrrrs@r`rrsP ""!" %$#$#%*%&!!!+/"$''#%<<+/-dSdSdS dS  dS  dSdSdSdS!dSdS#dS #dSdSdS !dS""%#dS$ %dS&!'dS($C=)dSdSR,004# || 5<<(   -      rbrc[0^\rSrSrSrSSSSSSS S S S S S S S S SSSSSSS\SS S S S S SSSSS S S SSSSS S SSSS4.S\\\\\44S\\\\\44S\S\S\ S\S\S\S\ S\ S\ S \ S!\S"\ S#\ S$\ S%\ S&\ S'\ S(\ S)\ S*\ S+\ S,\ \ S-\ S.\S/\ S0\ S1\ S2\ \ S3\ S4\ S5\ S6\ S7\ S8\ S9\ \ S:\ \ S;\ \S<\ \ S=\ S>\ S?\ \\\\4\4S@\ 4XU4SAjjjrSeSB\ 4SCjjrSfSEjrSeSF\R$SB\ SDS4SGjjr\R*R,SD\\ 4SHj5r\R*R,SeSI\ SDS4SJjj5r\R*R,SgSK\ SD\\ \44SLjj5r\R*R,SD\R$4SMj5rShS\S\ \ SDS4SNjjrSiS\ \\\4S\ \\\4SDS4SOjjrSD\\R@\ \R@44SPjr!SjSQ\R@SR\ \\\"\4SS\ ST\ SU\ SV\ SW\ SX\ \R@SY\ SD\\"\R@\\R@\"\R@444SZjjr#SkSR\\\"\4S[\ S\\ 4S]jjr$ShSQ\R@S^\ \ SD\R@4S_jjr%SlSQ\R@SX\ \R@SY\ SD\R@4S`jjr&SgSQ\R@Sa\ SD\R@4Sbjjr'SlSQ\R@SX\ \R@SY\ SD\R@4Scjjr(Sdr)U=r*$)mr+iaEva Vision Transformer w/ Abs & Rotary Pos Embed This class implements the EVA and EVA02 models that were based on the BEiT ViT variant * EVA - abs pos embed, global avg pool * EVA02 - abs + rope pos embed, global avg pool, SwiGLU, scale Norm in MLP (ala normformer) rAavg TrFrrrvNrzijg@gMbP?img_size patch_sizein_chans num_classes global_pool embed_dimdepthr1r2r3rrrrrr drop_rate pos_drop_ratepatch_drop_rateproj_drop_rateattn_drop_ratedrop_path_rater9r class_tokennum_reg_tokensno_embed_classuse_abs_pos_embuse_rot_pos_emb rope_typerope_grid_offsetrope_grid_indexingrope_temperaturerope_rotate_half use_post_normuse_pre_transformer_normuse_post_transformer_norm use_fc_normattn_pool_num_headsattn_pool_mlp_ratiodynamic_img_sizedynamic_img_padref_feat_shapehead_init_scalec/  >[T;U]5 U-U.S.n/US;deX@lX0lXPlU=Ul=UlUlU(aSOSU-UlUUl U)Ul SUl U$n0U&bU&n1OUS:Hn1U%bU%n2OU1(+n20n3U)(aU3R[SSS 95 [S+UUUUU*U$(+S .U3DU/D6UlURR n4[#URS 5(aURR%5OUn5U(a-[&R("[*R,"SSU40U/D65OSUlU(a-[&R("[*R,"SUU40U/D65OSUlU=(a UR0SLUlU(aU4OU4UR-n6U(a-[&R("[*R,"SU6U40U/D65OSUl[&R6"US 9UlUS:a[;UURS 9UlOSUlSUlU(aU+b [AU+5OSn+[S+UUU)(aSOURRBU!U S.U/D6n7US:Xa U7R[US95 SUlO US:XaU7R[SUU+S95 [ES+SU0U7D6Ul#OSUl#U0(a U"U40U/D6O[&RH"5Ul%[MUU5n8U#(a[NO[Pn9[&RR"[UU5V:s/sHKn:U9"S+0SU_SU_SU _SU _SU _SU _SU _SU_SU_SU_SU"_S UR_S!U_S"U_S#U8U:_S$U_S%U_U/D6PMM sn:5Ul+[UU5V:s/sHn:[S&U:3UU5S'9PM sn:Ul,U2(a U"U40U/D6O[&RH"5Ul-US(:XaA[]UR4U'=(d UU(=(d U U[&R^S).U/D6Ul0OSUl0U1(a U"U40U/D6O[&RH"5Ul1[&R6"U5Ul2US:a[&Rf"Xd40U/D6O[&RH"5Ul4U,Ul5URmSS*9 gs sn:fs sn:f),a Initialize the EVA Vision Transformer model. Args: img_size: Input image size (single int for square, or tuple for rectangular) patch_size: Patch size to divide image into tokens (single int for square, or tuple) in_chans: Number of input image channels num_classes: Number of classes (output dim) for classification head (final projection), 0 for pass-through global_pool: Type of global pooling for final sequence ('avg', 'token', 'map', etc.) embed_dim: Embedding dimension for tokens depth: Number of transformer blocks num_heads: Number of attention heads qkv_bias: Enable bias for query, key, value projections qkv_fused: Use a single projection for query, key, value mlp_ratio: Ratio of mlp hidden dim to embedding dim swiglu_mlp: Use SwiGLU activation in MLP scale_mlp: Apply scaling normalization in MLP (normformer style) scale_attn_inner: Apply scaling normalization inside attention attn_type: Type of attention module to use drop_rate: Dropout rate after final projection and pooling pos_drop_rate: Dropout rate for positional embeddings patch_drop_rate: Rate of dropping patches during training proj_drop_rate: Dropout rate for projections attn_drop_rate: Dropout rate for attention drop_path_rate: Stochastic depth rate norm_layer: Normalization layer constructor init_values: Initial layer-scale values class_token: Use class token num_reg_tokens: Number of additional learnable 'register' tokens to add to the sequence no_embed_class: Don't include position embeddings for class (or reg) tokens use_abs_pos_emb: Use absolute (learned) positional embeddings use_rot_pos_emb: Use rotary position embeddings rope_type: Type of RoPE to use ('cat', 'mixed', 'dinov3', etc.). rope_grid_offset: Offset for rotary position embedding grid rope_grid_indexing: Indexing mode for rotary position embeddings ('ij' or 'xy') rope_temperature: Temperature parameter for ROPE frequency computation rope_rotate_half: Use half rotation layout (rotate D/2 dims), else use interleaved rotation layout use_post_norm: Use post-norm transformer block type use_pre_transformer_norm: Use normalization layer before transformer blocks use_post_transformer_norm: Use normalization layer after transformer blocks use_fc_norm: Use normalization layer after pooling, before final classifier attn_pool_num_heads: Number of heads in attention pooling attn_pool_mlp_ratio: MLP ratio in attention pooling dynamic_img_size: Support dynamic image sizes in forward pass dynamic_img_pad: Apply dynamic padding for irregular image sizes ref_feat_shape: Reference feature shape for rotary position embedding scale head_init_scale: Initialization scale for classification head weights r>)ravgmaxmaxtokenmapr%rFNrNHWC)strict_img_size output_fmt)rrrrrrB feat_ratio)r)r5)r0r1 feat_shape temperature grid_indexingmixed)rTrz) in_pixels grid_offsetrrr0r1r2r3rrrrrrr<r5r7r6rr9rblocks.)modulenum_chs reductionr )r1rr9r needs_resetr)7rErFrrr num_featureshead_hidden_sizerr5rrgrad_checkpointingupdatedictr patch_embed num_patcheshasattrrrIrOrPrQ cls_token reg_token cls_embed pos_embedrXpos_dropr patch_drop rope_mixedr grid_sizerrlrUnorm_prerrr ModuleListrangeblocks feature_inforYr$r attn_poolfc_norm head_droprJheadr init_weights)'4#(mm*5\+%#*")   # " $   $   &  !0 $ "2 $ - #'"8"8 ) ) a&  &! "(% ("+%#$ .QVV[P\^P\1D'! yA FP\^4FJy/B/2;;= % 0-:-:%''   DN"DN6Fz)2r2BKKM I.?JQBIIi;;TVT_T_Ta . e,Y%#,^s AS 5S%rcUR[URUS95 URb[ URSS9 UR b[ UR SS9 UR b[ UR SS9 UR5 UR(a[UR[R5(a[ URRSS9 [R"5 URRR!UR5 URR"R!UR5 SSS5 ggg!,(df  g=f)Nr{Gz?std)applyr _init_weightsr&rr#r$fix_init_weightr isinstancer3rIrJrprPno_gradmul_rB)r\rs r`r4Eva.init_weightss 74--;GH >> % $..c 2 >> % $..c 2 >> % $..c 2    Jtyy"))$D$D $))** 4   %%d&:&:; ##D$8$89!%E  s AE.. E<rc[R"5 [UR5Hup[R "SUS--5nUR RRRU5 URRRRU5 M SSS5 g!,(df  g=f)z=Fix initialization weights by rescaling based on layer depth.g@r%N) rPrH enumerater.mathsqrtrrZrpdiv_rfc2)r\layer_idlayerrHs r`rFEva.fix_init_weightsy ]]_#,T[[#9 #A"67 &&++E2 $$))%0$:__s BB88 Cmc8[U[R5(aL[URSS9 UR b*[R RUR 5 ggU(a([US5(aXLaUR5 gggg)zInitialize weights for Linear layers and call reset_parameters on modules. Args: m: Module to initialize. needs_reset: Whether to call reset_parameters() on modules. rArBNr[) rGrIrJrrprBrdrer"r[)r\rTrs r`rEEva._init_weightssr a # # !(( ,vv!qvv&" WQ(:;;   AN;[rbcSS1n[USS5=n(a7[US5(a&XR5Vs1sHnSU3iM sn-$U$s snf)z(Parameters to exclude from weight decay.r&r#rlNno_weight_decayzrope.)rr"rX)r\nwdrlrs r`rXEva.no_weight_decayscK(D&$/ /D /WTCT5U5U/C/C/EF/E!E!+/EFF F GsAenablecXlg)z)Enable or disable gradient checkpointing.N)r)r\r[s r`set_grad_checkpointingEva.set_grad_checkpointings #)rbcoarsec[SSS/S9nU$)z(Create layer groupings for optimization.z ^cls_token|pos_embed|patch_embed)z^blocks\.(\d+)N)z^norm)i)stemr.)r)r\r_matchers r` group_matcherEva.group_matcher"s!4-/CD rbcUR$N)r3rgs r`get_classifierEva.get_classifier+s yyrbcXlUbX lUS:a'[R"URU5Ulg[R "5Ulg)zsReset the classifier head. Args: num_classes: Number of output classes. global_pool: Global pooling type. Nr)rrrIrJrrUr3)r\rrs r`reset_classifierEva.reset_classifier/sB'  "* >IAoBIIdnnk: SUS^S^S` rbc &URRnURRXS9 URbUR(aSO UR nURR U-nXPRRS:waD[R"[URURRUUSS95UlURb0URRURR5 gg)zUpdate the input image resolution and patch size. Args: img_size: New input resolution, if None current resolution is used. patch_size: New patch size, if None existing patch size is used. )rrNrr%T)new_sizeold_sizer5verbose) r r*set_input_sizer&rr5r!ryrIrOrrlupdate_feat_shape)r\rrprev_grid_sizer5num_new_tokenss r`rpEva.set_input_size;s))33 '''Q >> %%)%8%8d>T>T !--99+>aDDZDZ  ! q"a AAEAVDII//qf/=\`MI59YY5JDII//1PTM >> % MM$..// BC D >> % MM$..// BC D   $MIIfsl2IIfsl2$M MM!  ?? &"ooa0OA(\-E 6q V 4u55$1$;$;Aq$AM %2$;$;A$>Mrbrindicesreturn_prefix_tokensrY stop_earlyr intermediates_onlyrmrnc `US;dS5eUS:Hn /n [[UR5U5upURupnnUR U5nUR U5unnUR U5n[RR5(dU(d URnOURSU S-n[USS5(aUb[U5HunnUR(a3[RR5(d[UUUUXS9nO U"UUUXS9nUU ;dM]U RU(aURU5OU5 M O[U5HunnUR(a0[RR5(d [UUUXS9nOU"UUXS9nUU ;dMWU RU(aURU5OU5 M UR (aJU Vs/sHnUSS2S UR 24PM nnU Vs/sHnUSS2UR S24PM n nU (adURR#UU45unnU Vs/sH7nUR%UUUS 5R'S S SS 5R)5PM9 n n[RR5(dU(a[+[-U W55n U(aU $URU5nX4$s snfs snfs snf) aeForward features that returns intermediates. Args: x: Input image tensor indices: Take last n blocks if an int, if is a sequence, select by matching indices return_prefix_tokens: Return both prefix and spatial intermediate tokens norm: Apply norm layer to all intermediates stop_early: Stop iterating over blocks when last desired intermediate hit output_fmt: Shape of intermediate feature outputs intermediates_only: Only return intermediate features attn_mask: Optional attention mask for masked attention is_causal: If True, use causal (autoregressive) masking in attention )NCHWNLCz>Output format for EVA-ViT features must be one of NCHW or NLC.rNr%r)FrrrqrArr)r'lenr.ryr rr+rPr is_scriptingrrLrr(rxrYr5dynamic_feat_sizer}r~ contiguouslistzip)r\rrrrYrr rrmrnr} intermediates take_indices max_indexr_heightwidthr}r.r>blky prefix_tokensr{r|s r`forward_intermediatesEva.forward_intermediatess0_,n.nn,& "6s4;;7G"Q  ggfe   Q ??1-= MM!  99 ! ! # #:[[F[[)a-0F 4u - --2K#F+3**5993I3I3K3K"3 a0@IkAAM!$4 _A $!((11E ,$F+3**5993I3I3K3K"3 hAAMY\A $!((11E ,  ! !ERS]Qq!D$:$:"::;]MSDQRMqQq$"8"8"99:MMR ##55vuoFDAq^kl^kYZQYYq!Q3;;Aq!QGRRT^kMlyy%%'',@ ]M!BCM  IIaLTRmsL!9L&>L+ prune_norm prune_headc,[[UR5U5upEURSUS-UlU(a[R"5UlU(a3SUl[R"5UlURSS5 U$)z?Prune layers not required for specified intermediates. Nr%rr) r'rr.rIrUrYr0r1rj)r\rrrrrs r`prune_intermediate_layersEva.prune_intermediate_layersso#7s4;;7G"Q kk.9q=1  DI !DN;;=DL  ! !!R (rb pool_typecURbURU5nU$Uc UROUn[XURS9nU$)N)rr5)r0rrr5)r\rrs r`poolEva.poolsH >> %q!AH(1(9D$$y AdF\F\ ]rbc JURU5nURU5upURU5n[USS5(aoUbl[ UR 5HRupVUR (a2[RR5(d[XaXEX#S9nMHU"XUX#S9nMT O[UR HKnUR (a0[RR5(d [XaXBUS9nMDU"XX#S9nMM URU5nU$)zForward pass through feature extraction layers. Args: x: Input tensor. attn_mask: Optional attention mask for masked attention is_causal: If True, use causal (autoregressive) masking in attention. Returns: Feature tensor. r)Fr) r rr+rrLr.rrPrrr(rY)r\rrmrnr}r>rs r`forward_featuresEva.forward_featuress   Q ??1- MM!  4u - --2K$DKK0**5993I3I3K3K"3 0@IkAA!$4 _A 1{{**5993I3I3K3K"3 ^ghAAY\A # IIaLrb pre_logitscURU5nURU5nURU5nU(aU$URU5$)zForward pass through classifier head. Args: x: Feature tensor. pre_logits: Return pre-logits if True. Returns: Output tensor. )rr1r2r3)r\rrs r` forward_headEva.forward_headsA IIaL LLO NN1 q0DIIaL0rbcHURXUS9nURU5nU$)zForward pass. Args: x: Input tensor. attn_mask: Optional attention mask for masked attention is_causal: If True, use causal (autoregressive) masking in attention. Returns: Output tensor. )rmrn)rr)r\rrmrns r`r Eva.forward,s.  ! !!I ! N   a rb)r0r.r%r#rrr1r/rrr3r2rrrrrYr+rrr5r(r r'r&r$rlr))TrFrf)NN)NFFFrFNF)r%FT)NF)+rrrrrrr rr rrrrrrFr4rFrIModulerErPrignorer rXr]rrrcrgrjrprrrrrrrrrrrrs@r`r+r+sG5868#$ !"!$#$#%*"!#%%'$&$&$&#,+/ $"##($($)',&(&*&,%*"'-28<*.1537%*$)DH%*__-CsCx01_-c5c?23_- _-  _-  _-_-_-_-_-_-_-_-!_-_- #!_-"#_-$%_-&!'_-(#)_-*"+_-,"-_-."/_-0!1_-2"%3_-45_-6 7_-8!9_-:";_-<"=_-> }?_-@$A_-B!$C_-D$E_-F#G_-H I_-J'+K_-L(0~M_-N"$O_-P"*#Q_-R"*%S_-T#U_-V"W_-X%U5c?C+?%@AY_-Z#[_-_-B::"1 !ryy !t !t ! YYS YY)T)T)) YYDT#s(^ YY  aC ahsm aW[ a3748DuS#X/D!sCx1D  D:5 uU\\8ELL3I%IJ5 t8<).$$',04#K ||K eCcN34K #' K  K  K K !%K   -K K  tELL!5tELL7I)I#JJ KK ^./$# 3S >* $ellx} 15# %||%  -% %  %N 1ell 1 1 1$15# ||  -   rb state_dictmodelprefixrcURSU5nUR5VVs0sHup4URSS5U_M nnn0n/SQn[U5nUR5GHup4U(aUR U5(dM%X7SnUHnURUSUS5nM UR S5(aURS S5nURS S 5nURS S 5nUR S5(a`UR SS-n UR S5(aUSU US'XISUS'O%UR S5(aUSU US'XISUS'GM OUS:Xa;SnURSS5n[R"UR S5US'O@US:Xa#SnURS5RS5nOUS:XaURS5nXEU'GM U$s snnf)zConvert Perception Encoder weights. Args: state_dict: State dictionary to convert. model: Target model instance. prefix: Prefix to strip from keys. Returns: Converted state dictionary. rzmodule.r) )conv1patch_embed.proj)positional_embeddingr&)ztransformer.resblocks.r)ln_prer+)ln_postrY)ln_rY)z ls_1.gammar)z ls_2.gammar)in_proj_zqkv.)out_projrZ)zmlp.c_fcmlp.fc1)z mlp.c_projmlp.fc2Nrr%r0zattn_pool.attnzattn_pool.layernormzattn_pool.normzattn_pool.probezattn_pool.latentz attn_pool.qkvrArpzattn_pool.q.weightzattn_pool.kv.weightrBzattn_pool.q.biaszattn_pool.kv.biasrZz head.weightz head.biasclass_embeddingr#r&) getitemsreplacer startswithryendswithrrPzerosrz) rrrrrout_dictswaps len_prefixspr0s r` _convert_perAs4J:D:J:J:LM:L$!!))Ir*A-:LJMH EVJ  " <<''+AB "Q%A'A << $ $ *K8A /1ABA +-?@A||O,,ggajAo::h''56tWH1267gH23ZZ''34Tc7H/045dGH01-&[A Aq!A$)KK $;H[ ! # #A A((+A +  AA ?#B OgNsH interpolation antialiasc x0nURSU5nURSU5nURSU5nURSU5nSU;a [X5$SU;a [XSS9$S U;aS nO S U;aS nOSnS U;nU(+=(a US-U;nUS-U;n[U5n UR5GHupU(aU R U5(dM%XSn SU ;aU S:XdM7U(a[ SV s/sHoR U 5PM sn 5(aMoU R S5(aMU R S5(aRU RSS5n URU 5 U RSSS9unnnXU 'UXJRSS5'MU RSS5n U RSS5n U RS S5n O/U(a(U S ;a"U S!:XdU S":XaU RS#S$5n OGMUS%U ;aeURRRRu nnnU RSU:wdU RS&U:wa[!U UU4UUS'S(9n OsU S):XamU RS*UR"RS*:waC[%US+S,5(aS-O [%US.S*5n['U URR(UUUS'S/9n U RS0S15n U RS2S35n U RS4S55n U RS6S75n U RS8S95n U RS:S;5n U(a$U RSS<5n U RSS=5n XU 'GM U$s sn f![an[U5 SnAGMSnAff=f)>a:Convert patch embedding weight from manual patchify + linear proj to conv. Args: state_dict: Checkpoint state dictionary. model: Target model instance. interpolation: Interpolation method for resizing. antialias: Whether to use antialiasing when resizing. Returns: Filtered state dictionary. model_emarrrzvisual.conv1.weightz conv1.weightr)rzvisual.trunk.pos_embedz visual.trunk.zvisual.pos_embedvisual.storage_tokens mask_tokenzblocks.0.attn.q_proj.weightNrlz rope.freqs)z.periodsz .bias_maskrlocal_cls_normzqkv.biasrRrArqrurTz ls1.gammarz ls2.gammarr$)rzlm_head.weightz lm_head.bias norm.weight norm.biasrrrYr1zpatch_embed.proj.weightrxT)rrror&r%rFrr5)rmr5rrroz mlp.ffn_lnzmlp.normzattn.inner_attn_lnz attn.normzmlp.w12rzmlp.w1z mlp.fc1_gzmlp.w2z mlp.fc1_xzmlp.w3rz q_proj.biasz v_proj.bias)rrrrranyrr get_parameter Exceptionprintchunkr rZrpryrr&rrr*)rrrrrrdinov3_weights mim_weightsno_qkvrrrfq_bias_keqvkvvvrr{r|r5s r`checkpoint_filter_fnrs"H Z8J4J*5J j9J *:-- : %:R88 :-  z )%3N$$L,)>*)LK 3 3z AFVJ  " <<''+A Q;qL0  +ST+SaJJqM+STUU||,--zz*%%99Z:''1 WWQBW/ B%'"<>:x89 +y1A +y1A *K8A Q"nnM!Q+%5IIfi0 $ )**//66<r)eva02_large_patch14_448r!r"s r`rHrHr@rbc p[SSSSSURSS5S9n[S S U0[U40UD6D6nU$) z?EVA-g CLIP model (only difference from non-CLIP is the pooling)rrrrrrr )rrrr1rrr)eva_giant_patch14_clip_224rrrr"s r`rJrJsLR2JJ}g68J  j jtT^OibhOi jE Lrbc ~[SSSSSSSSSSSSURS S 5S 9 n[S S U0[U40UD6D6nU$)zUAn EVA-CLIP specific variant that adds additional attn scale layer-norm to eva02_baserrrrFr,Tr-rr ) rrrrr1r3rrrrrrrr)eva02_base_patch16_clip_224rKr"s r`rMrMsfJJ}g6J  k* kPTU_PjciPj kE Lrbc ~[SSSSSSSSSSSS URS S 5S 9 n[SS U0[U40UD6D6nU$)VAn EVA-CLIP specific variant that adds additional attn scale layer-norm to eva02_largerrrr=rr,FTr-rr  rrrrr1rr3rrrrrrr)eva02_large_patch14_clip_224rKr"s r`rQrQfJJ}g6J  l: lQUV`QkdjQk lE Lrbc ~[SSSSSSSSSSSS URS S 5S 9 n[SS U0[U40UD6D6nU$)rOrrrr=rr,FTr-rr rPr)eva02_large_patch14_clip_336rKr"s r`rTrTrRrbc t[SSSSSSSURSS 5S 9n[S S U0[U40UD6D6nU$) zCAn EVA-CLIP specific variant that uses residual post-norm in blocksrri@rgI$I$!@Trr )rrrrr1rrrr)eva02_enormous_patch14_clip_224rKr"s r`rWrWsWJJ}g6 J  oj oTXYcTngmTn oE Lrbc ^[SSSSSSSSSS SSS S 9 n[S S U0[U40UD6D6nU$)timm SBB ViT with ROPEr rrrrTh㈵>Fr%r- rrrrr1r3r2rrrrrrr)$vit_medium_patch16_rope_reg1_gap_256r!r"s r`r\r\s[J  t: tY]^hYslrYs tE Lrbc ^[SSSSSSSSSS SSS S 9 n[S S U0[U40UD6D6nU$)rYr rrrTFrZr%r-r[r)%vit_mediumd_patch16_rope_reg1_gap_256r!r"s r`r_r_s[J  uJ uZ^_iZtmsZt uE Lrbc ^[SSSSSSSSSS SSS S 9 n[S S U0[U40UD6D6nU$)rYr rir TrZFrsr-r[r)%vit_betwixt_patch16_rope_reg4_gap_256r!r"s r`rbrb/s[J  uJ uZ^_iZtmsZt uE Lrbc ^[SSSSSSSSSSSSS S 9 n[S S U0[U40UD6D6nU$) rYr rrrTrZFr%r-r[r)"vit_base_patch16_rope_reg1_gap_256r!r"s r`rdrdEs[J  r rW[\fWqjpWq rE Lrbc v[SSSSSSSSSS S S S S[[S S9S9n[SSU0[U40UD6D6$)HPerception Encoder (PE) ViT from Meta (https://arxiv.org/abs/2504.13181)rr+rrArr rlTr=r=rxyrrZepsrrrr1rrrrrrrrrrr9r)vit_pe_core_tiny_patch16_384rrrrrrr#s r`rlrl[sd!%9$/J$  k* kPTU_PjciPj kkrbc v[SSSSSSSSSS S S S S[[S S9S9n[SSU0[U40UD6D6$)rfrrrr3rr rlTrgrrhrrZrirkr)vit_pe_core_small_patch16_384rmrns r`rprptsd!%9$/J$  l: lQUV`QkdjQk llrbc v[SSSSSSSSSSS S S S[[S S 9S9n[SSU0[U40UD6D6$)rfrrrrr rlT)rrrrhrrZrirkr)vit_pe_core_base_patch16_224rmrns r`rrrrsd!%9$/J$  k* kPTU_PjciPj kkrbc v[SSSSSSSSSS S S S S[[S S9S9n[SSU0[U40UD6D6$)rfrrr=rrr rlTrgrrhrrZrirkr)vit_pe_core_large_patch14_336rmrns r`rtrtsd!%9$/J$  l: lQUV`QkdjQk llrbc v[SSSSSSSSS S S S S S [[SS9S9n[SSU0[U40UD6D6$)rfr2rUUUUUU@r rlFT rzrhrrrZri)rrrr1rrrrrrrrrrr9r) vit_pe_core_gigantic_patch14_448rmrns r`r{r{sd!%9$/J$  oj oTXYcTngmTn oorbc [S 0SS_SS_SS_SS_S S _S S _S S_SS_SS_SS_SS_SS_SS_SS_SS_S[[SS9_6n[S!SU0[U40UD6D6$)"rfrrrrrr1rrrrrlrTrrryrrrrhrrFrr皙?r9rZrirr)vit_pe_lang_large_patch14_448rmrns r`rrs       "&#( 9$/!J&  l: lQUV`QkdjQk llrbc v[SSSSSSSSS S SSSS [[S S 9S9n[SSU0[U40UD6D6$)rfrrv/rrxrlFTryrhr~rZrirrrr1rrrrrrrrrrr9r) vit_pe_lang_gigantic_patch14_448rmrns r`rrsd!%"'9$/J$  oj oTXYcTngmTn oorbc t[SSSSSSSSSSS S S [[S S 9S9n[SSU0[U40UD6D6$)rfrr+rrArrlTFryrrhrZrirrrr1rrrrrrrrrr9r)vit_pe_spatial_tiny_patch16_512rmrns r`rr sa!%"'9$/J"  nZ nSWXbSmflSm nnrbc t[SSSSSSSSSSS S S [[S S 9S9n[SSU0[U40UD6D6$)rfrrrr3rrlTFryrrhrZrirr) vit_pe_spatial_small_patch16_512rmrns r`rr sa!%"'9$/J"  oj oTXYcTngmTn oorbc t[SSSSSSSSSSSS S [[S S 9S 9n[SSU0[U40UD6D6$)rfrrrrrlTFryrrhrZrirr)vit_pe_spatial_base_patch16_512rmrns r`rr3 sa!%"'9$/J"  nZ nSWXbSmflSm nnrbc t[SSSSSSSSSSS S S [[S S 9S9n[SSU0[U40UD6D6$)rfrrr=rrrlTFryrrhrZrirr) vit_pe_spatial_large_patch14_448rmrns r`rrJ sa!%"'9$/J"  oj oTXYcTngmTn oorbc v[SSSSSSSSS S SSSS [[S S 9S9n[SSU0[U40UD6D6$)rfrrvrwrrxrlFTryrhr~rZrirr)#vit_pe_spatial_gigantic_patch14_448rmrns r`rra sd!%"'9$/J$  r rW[\fWqjpWq rrrbc `[SSSSSSSSSS S SS S S 9n[SSU0[U40UD6D6nU$)z=RoPE-Axial ViT-S/16 from https://github.com/naver-ai/rope-vitrrrr3rsrlTrZr FrhY@rrrr1rrr2rrrrrrrr)vit_small_patch16_rope_224r!r"s r`rrz s]J  j jtT^OibhOi jE Lrbc b[SSSSSSSSSSS SSS S S 9n[SS U0[U40UD6D6nU$)z=RoPE-Axial ViT-B/16 from https://github.com/naver-ai/rope-vitrrrrsrlFTrZr rhr)rrrr1rrrr2rrrrrrrr)vit_base_patch16_rope_224r!r"s r`rr s`J"  i idS]NhagNh iE Lrbc `[SSSSSSSSSSS SS S S 9n[SS U0[U40UD6D6nU$)z=RoPE-Axial ViT-L/16 from https://github.com/naver-ai/rope-vitrrr=rsrlTrZr Frhrrr)vit_large_patch16_rope_224r!r"s r`rr s]J  j jtT^OibhOi jE Lrbc b[SSSSSSSSSS S SS S S S9n[SSU0[U40UD6D6nU$)z=RoPE-Mixed ViT-S/16 from https://github.com/naver-ai/rope-vitrrrr3rsrlTrZr Frh$@rrrrr1rrr2rrrrrrrrr) vit_small_patch16_rope_mixed_224r!r"s r`rr saJ"  pz pUYZdUohnUo pE Lrbc b[SSSSSSSSSSS SS S S S 9n[SSU0[U40UD6D6nU$)z=RoPE-Mixed ViT-B/16 from https://github.com/naver-ai/rope-vitrrrrsTrlrZr Frhrr)rrrr1rr2rrrrrrrrrr)vit_base_patch16_rope_mixed_224r!r"s r`rr saJ"  oj oTXYcTngmTn oE Lrbc b[SSSSSSSSSSS SS S S S 9n[SSU0[U40UD6D6nU$)z=RoPE-Mixed ViT-L/16 from https://github.com/naver-ai/rope-vitrrr=rsrlTrZr Frhrrrr) vit_large_patch16_rope_mixed_224r!r"s r`rr saJ"  pz pUYZdUohnUo pE Lrbc b[SSSSSSSSSS SSSS S S 9n[SS U0[U40UD6D6nU$)zCRoPE-Axial + APE ViT-S/16 from https://github.com/naver-ai/rope-vitrrrr3rsrlTrZr rhrrrrr1rrr2rrrrrrrrr)vit_small_patch16_rope_ape_224r!r"s r`rr saJ"  nZ nSWXbSmflSm nE Lrbc b[SSSSSSSSSSSSSS S S 9n[S S U0[U40UD6D6nU$)zCRoPE-Axial + APE ViT-B/16 from https://github.com/naver-ai/rope-vitrrrrsrlTrZr rhrrr)vit_base_patch16_rope_ape_224r!r"s r`rr! saJ$  mJ mRVWaRlekRl mE Lrbc b[SSSSSSSSSSSSSS S S 9n[S S U0[U40UD6D6nU$)zCRoPE-Axial + APE ViT-L/16 from https://github.com/naver-ai/rope-vitrrr=rsrlTrZr rhrrr)vit_large_patch16_rope_ape_224r!r"s r`rr: saJ$  nZ nSWXbSmflSm nE Lrbc [S0SS_SS_SS_SS_S S _S S _S S_SS_SS_SS_SS_SS_SS_SS_SS_SS_6n[SSU0[U40UD6D6nU$) zCRoPE-Mixed + APE ViT-S/16 from https://github.com/naver-ai/rope-vitrrrrrrr1r3rrsrrlr2TrrZrrr rrrrrhrrrrrr)$vit_small_patch16_rope_mixed_ape_224r!r"s r`rrS s       !J&  t: tY]^hYslrYs tE Lrbc [S0SS_SS_SS_SS_SS _S S _S S _SS_SS _SS_SS _SS _SS _SS_SS_SS_6n[SSU0[U40UD6D6nU$)zCRoPE-Mixed + APE ViT-B/16 from https://github.com/naver-ai/rope-vitrrrrrrr1rrsrrlr2TrrZrrr rrrrrhrrrrrr)#vit_base_patch16_rope_mixed_ape_224r!r"s r`rrm s       !J$  s* sX\]gXrkqXr sE Lrbc [S0SS_SS_SS_SS_SS _S S _S S _SS_SS _SS_SS _SS _SS _SS_SS_SS_6n[SSU0[U40UD6D6nU$)zCRoPE-Mixed + APE ViT-L/16 from https://github.com/naver-ai/rope-vitrrrrrr=r1rrsrrlr2TrrZrrr rrrrrhrrrrrr)$vit_large_patch16_rope_mixed_ape_224r!r"s r`rr s       !J$  t: tY]^hYslrYs tE Lrbc z[SSSSSSSSS SSSS S[[SS 9S 9n[SS U0[U40UD6D6nU$)zDINOv3 S/16 https://arxiv.org/abs/2508.10104 NOTE: Pass global_pool='token' to use CLS-token pooling (matches upstream DINOv3). rTrrr3FrZdinov3drsrirrrrr1r2rrrrrrrrr9r)vit_small_patch16_dinov3rmr"s r`rr sg 9$/!J$  hz hTR\Mg`fMg hE Lrbc z[SSSSSSSSSSSS S S [[SS 9S 9n[SS U0[U40UD6D6nU$)zDINOv3 S/16 w/ QKV bias enabled (but zero) https://arxiv.org/abs/2508.10104 NOTE: Pass global_pool='token' to use CLS-token pooling (matches upstream DINOv3). rTrrr3rZrrFrsrirr)vit_small_patch16_dinov3_qkvbrmr"s r`rr sh 9$/!J$  mJ mRVWaRlekRl mE Lrbc [S0SS_SS_SS_SS_S S _S S _S S_SS_SS_SS_SS_SS _SS_SS_SS_SS _S[[SS9_6n[S SU0[U40UD6D6nU$)!zDINOv3 S/16 Plus https://arxiv.org/abs/2508.10104 NOTE: Pass global_pool='token' to use CLS-token pooling (matches upstream DINOv3). rrrTrrrrr1r3r2FrrZrrrrrrrrrrrrsrr9rirr)vit_small_plus_patch16_dinov3rmr"s r`rr s       !"#$9$/%J(  mJ mRVWaRlekRl mE Lrbc [S0SS_SS_SS_SS_S S _S S_S S _SS_SS_SS_SS_SS_SS_SS_SS_SS_S[[S S9_6n[S SU0[U40UD6D6nU$)!zDINOv3 S/16 Plus w/ QKV bias enabled (but 0) https://arxiv.org/abs/2508.10104 NOTE: Pass global_pool='token' to use CLS-token pooling (matches upstream DINOv3). rrrTrrrrr1r3r2rrZrrrrrrrFrrrrrsrr9rirr)"vit_small_plus_patch16_dinov3_qkvbrmr"s r`rr s       !"#$9$/%J(  r rW[\fWqjpWq rE Lrbc z[SSSSSSSSSSSSS S[[SS 9S 9n[S S U0[U40UD6D6nU$)zDINOv3 B/16 https://arxiv.org/abs/2508.10104 NOTE: Pass global_pool='token' to use CLS-token pooling (matches upstream DINOv3). rTrrFrZrrrsrirr)vit_base_patch16_dinov3rmr"s r`rr sg 9$/!J$  gj gDQ[Lf_eLf gE Lrbc z[SSSSSSSSSSSSS S[[SS 9S 9n[S S U0[U40UD6D6nU$)zDINOv3 B/16 w/ QKV bias enabled (but zero) https://arxiv.org/abs/2508.10104 NOTE: Pass global_pool='token' to use CLS-token pooling (matches upstream DINOv3). rTrrrZrrFrsrirr)vit_base_patch16_dinov3_qkvbrmr"s r`rr* sh 9$/!J$  l: lQUV`QkdjQk lE Lrbc z[SSSSSSSSSSSSS S[[SS 9S 9n[S S U0[U40UD6D6nU$)zDINOv3 L/16 https://arxiv.org/abs/2508.10104 NOTE: Pass global_pool='token' to use CLS-token pooling (matches upstream DINOv3). rTrr=FrZrrrsrirrrrr1r2rrrrrrrrr9r)vit_large_patch16_dinov3rmr"s r`rrE sg 9$/!J$  hz hTR\Mg`fMg hE Lrbc z[SSSSSSSSSSSSS S[[SS 9S 9n[S S U0[U40UD6D6nU$)zDINOv3 w/ QKV bias enabled (but zero) https://arxiv.org/abs/2508.10104 NOTE: Pass global_pool='token' to use CLS-token pooling (matches upstream DINOv3). rTrr=rZrrFrsrirr)vit_large_patch16_dinov3_qkvbrmr"s r`rr` sh 9$/!J$  mJ mRVWaRlekRl mE Lrbc [S0SS_SS_SS_SS_S S _S S _S S_SS_SS_SS_SS _SS_SS_SS_SS_SS _S[[SS9_6n[S SU0[U40UD6D6nU$)!zDINOv3 H/16 Plus https://arxiv.org/abs/2508.10104 NOTE: Pass global_pool='token' to use CLS-token pooling (matches upstream DINOv3). rrrTrrrrzr1r^r2FrrZrrrrrrrrrrrrsrr9rirr)vit_huge_plus_patch16_dinov3rmr"s r`rr{ s       !"#$9$/%J*  l: lQUV`QkdjQk lE Lrbc [S0SS_SS_SS_SS_S S _S S_S S _SS_SS_SS_SS_SS_SS_SS_SS_SS_S[[S S9_6n[S SU0[U40UD6D6nU$)!zDINOv3 H/16 Plus w/ QKV bias enabled (but zero) https://arxiv.org/abs/2508.10104 NOTE: Pass global_pool='token' to use CLS-token pooling (matches upstream DINOv3). rrrTrrrrzr1r^r2rrZrrrrrrFrrrrrrsrr9rirr)!vit_huge_plus_patch16_dinov3_qkvbrmr"s r`rr s       !"#$9$/%J*  q qVZ[eVpioVp qE Lrbc [S!0SS_SS_SS_SS_S S _S S _S S_SS_SS_SS_SS_SS _SS_SS_SS_SS_SS _S[[SS9_6n[S"S U0[U40UD6D6nU$)#zDINOv3 7B/16 https://arxiv.org/abs/2508.10104 NOTE: Pass global_pool='token' to use CLS-token pooling (matches upstream DINOv3). rrrTrirrr1rzr2FrrrrrZrrrrrrrrrrVrrsrr9rirr)vit_7b_patch16_dinov3rmr"s r`rr s      "#$%&9$/'J,  eJ e$zJd]cJd eE Lrb)r)rTr)r)rrMr functoolsrtypingrrrrrr r r rPtorch.nnrItorch.nn.functional functionalr{ timm.datar r rr timm.layersrrrrrrrrrrrrrrrrr r!r"r#r$_builderr& _featuresr' _manipulater( _registryr)r*__all__rr-rrr+rrrrrrrr r default_cfgsr r&r(r/r4r9r?rBrDrFrHrJrMrQrTrWr\r_rbrdrlrprrrtr{rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrbr`rsl=@ III dd,++#< 'a299aHAryyAHqryyqhy"))y~CS%,,&'C 99C C #u||  CR' yell*+yyyyy y  #u||  yx$S>cT#s(^(S#X(SDcN.%R&)$+R&)$ 3(+DR&/ "(< 3( 1DR&$/ "(< 3( 1D%R&25d 3(73R&<6t 3(8=R&F5d 3(7GR&T/ 31UR&^0 32_R&h/ 31iR&r0 32sR&|/ 31}R&J0 3(PU2KR&T1$ 3(PU3UR&^0 3(PU2_R&l')mR&v(*wR&@')AR&J(*KR&T')UR&b+D -cR&p*4 ,qR&~+D -R&L,T .MR&Z,T 3 .[R&j.t 0kR&x3D 5yR&F/1GR&P4T 4 /6QR&Z5d 4 /7[R&d5d 47eR&l24 4 /4mR&z&w! ({R&H'! )IR&V&w! (WR&d'! )eR&r*7! ,sR&B'! )CR&P.w! 0QR&^*7! ,_R&l1'! 3mR&|)'! +}R&J*7! ,KR&X)'! +YR&f*7! ,gR&t-g! /uR&F ,T "(<.G R&P +D "(<-Q R&Z ,T "(<.[ R&d 24 "(<4e R&n 1$ "(<3o R&x 24 "(<4y R&B 0 "(<2C R&L / "(<1M R&V 0 "(<2W R&` 6t "(<8a R&j 5d "(<7k R&t 6t "(<8u R&H (*I R&N -k/O R&T -k/U R&Z 2;4[ R&` ' )a R&f ,[.g R&l (*m R&r -k/s R&x '  "(=)y R&@ ,[ "(=.A R&H .9.3>3'2'&1 "(=&[ R&R j dddt#"3"t#&3&t#"3"t#&3&4cDs*T*T*  3   TPS*dQT*dQT*4c*lTlll0mdmmm.lTlll.mdmmm.ppCpp.mdmmm0ppCpp.oo3oo,ppCpp,oo3oo,ppCpp,sDssss04c,$S.4c,C.3.C0t#.d0t#0TPS2Ds0TPS0C4d4d84c834T4C4d4T:$S8drb