3j SrSSKrSSKJr SSKJrJrJrJrJ r J r J r J r SSK r SSKJr SSKJs Jr SSKJrJr SSKJrJrJrJrJr SSKJr SS KJr SS K J!r!J"r" SS K#J$r$J%r% SS K&J'r' S S/r("SS\RR5r*S\ RVS\ RV4Sjr,SRS\ RVS\ RVS\ RVS\-S\ RV4 Sjjr.SSS\ RVS\ RVS\ RVS\-S\-S\ RV4 Sjjr/"SS\RR5r0"SS\RR5r1"SS\RR5r2"S S!\RR5r3"S"S#\RR5r4"S$S%\RR5r5"S&S \RR5r6"S'S\RR5r7STS)\RRS*\8S+\9SS4S,jjr:STS-\8S+\9S\4S.jjr;S/\\8\ RV4S0\6S\\8\ RV44S1jrSUS4\8S5\9S\74S7jjr?SVS8\8S\\8\44S9jjr@\$"\@"S:S;SS;SXES.n[TU]5 X0l[R"X4SS0UD6UlU(aUR S[R"S0UD65 UR S[R"S0UD65 UR S[R"S0UD65 UR S[R"S0UD65 UR5 gg) NdevicedtypebiasF input_min input_max output_min output_max)) super__init__r"nnLinearlinearregister_buffertorchemptyreset_parameters)selfr r!r"r&r'dd __class__s P/home/wildlama/miniconda3/lib/python3.13/site-packages/timm/models/gemma4_vit.pyr/Gemma4ClippableLinear.__init__'s / &ii LLL   ekk.C.C D  ekk.C.C D  u{{/D/D E  u{{/D/D E  ! ! # cLUR(aURR[S5*5 URR[S55 UR R[S5*5 UR R[S55 gg)Ninf)r"r)fill_floatr*r+r,r7s r:r6&Gemma4ClippableLinear.reset_parameters;sn    NN %, / NN u . OO ! !5<- 0 OO ! !%, / r<xcUR(a+[R"XRUR5nUR U5nUR(a+[R"XR UR5nU$N)r"r4clampr)r*r2r+r,r7rCs r:forwardGemma4ClippableLinear.forwardDsV    A~~t~~>A KKN    A@Ar<)r2r")FNNr#N)__name__ __module__ __qualname____firstlineno____doc__intboolr/r6r4TensorrH__static_attributes__ __classcell__r9s@r:rr sh!& $$$ $ $$(0%,,r<rrCr#cUSSURSS-24nUSURSS-S24n[R"U*U4SS9$)z*Rotates half the hidden dims of the input..Ndim)shaper4cat)rCx1x2s r: rotate_halfr_MsZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r<cossin unsqueeze_dimclURU5nURU5nX-[U5U--$)z0Apply rotary position embedding to input tensor.) unsqueezer_)rCr`rarbs r:apply_rotary_pos_embreTs6 -- &C -- &C G A, --r<ndimc JURSnSUSU---nU/U-n[R"XSS9n[R"XSS9n [R"X'SS9n [U5V s/sHn [ XXXUS9PM n n [R "U SS9$s sn f)zApply multidimensional RoPE to input tensor. Splits input along head_dim into ndim parts, applies RoPE to each, then concatenates back. rWrXrY)rb)r[r4splitrangerer\) rCr`rarfrbnum_input_channelsnum_rotated_channels_per_dim split_sizesx_parts cos_parts sin_partsky_partss r:apply_multidimensional_roperr`s#$(:q4x(H#I /047Kkk!b1G C"5I C"5IlqqukvkvfgWZy|S`akv  99W" %%s-B c^\rSrSrSrSS\S\SS4U4SjjjrSSjrSS jr SS jr S \ RS \ RS\ \ R\ R44S jrSrU=r$)Gemma4RotaryEmbedding2Dzz2D Rotary Position Embedding for Gemma4 vision encoder. Computes RoPE independently for each spatial dimension (x, y), using theta=100.0 and the partitioned head_dim. Nhead_dim rope_thetar#c >[TU]5 XlX lUS-S-nUR S[ R "XS[ RS9SS9 UR5 g)NrXinv_freqr%F) persistent) r.r/rvrwr3r4r5r@ _init_buffers)r7rvrwr&r' num_freqsr9s r:r/ Gemma4RotaryEmbedding2D.__init__sc  $]q(    KK  D  r<c URS-nSUR[R"SUS[RUR R S9U- -- nUR RU5 g)z.Compute and fill non-persistent buffer values.rX?rr'r&N)rvrwr4aranger@ryr&copy_)r7 spatial_dimrys r:r{%Gemma4RotaryEmbedding2D._init_buffersscmmq(  OO Q Qekk$--J^J^_bmm o  H%r<c$UR5 grEr{rAs r:r6(Gemma4RotaryEmbedding2D.reset_parameterss r<c$UR5 g)z"Initialize non-persistent buffers.NrrAs r:init_non_persistent_buffers3Gemma4RotaryEmbedding2D.init_non_persistent_bufferss r<rC position_idscP[R"5 URSSS2S4R5R UR SSS5R UR5n/n/n[S5HnUSS2SS2U4nUSS2SSS24R5nUR5UR5-RSS5n [R"X4SS9n URU R55 URU R55 M [R"USS9R URS9n [R"USS9R URS9n SSS5 X4$!,(df  W W 4$=f)z Args: x: Hidden states tensor, used only for dtype/device. position_ids: (B, N, 2) Gemma4-internal ``(x, y)`` patch coords. Returns: cos, sin: (B, N, head_dim) tensors for RoPE application. NrrWrrXrY)r')r4no_gradryr@expandr[tor&ri transposer\appendr`rar') r7rCrinv_freq_expandedall_cosall_sinidim_posdim_pos_expandedfreqsembr`ras r:rHGemma4RotaryEmbedding2D.forwardse ]]_ $ dAtm < B B D K KLL^L^_`Lacegh i l lmnmumu v *,G*,G1X&q!Qw/#*1dA:#6#<#<#> *0025E5K5K5MMXXYZ\]^iiB7swwy)swwy)))G,//agg/>C))G,//agg/>C x!_ Cxs E3F F%rvrw)Y@NNrJ)rKrLrMrNrOrPr@r/r{r6rr4rRr rHrSrTrUs@r:rtrtzs!&   ,& ||  ,,  u||U\\) *  r<rtc ^\rSrSrSrSS\\\\\44S\S\S\SS4 U4S jjjrSS jr S \S \S \S\ RS\\ R\ R44 Sjr S\ RS\ RS\ R4SjrSS\\ R\\\ R44S\\ RS\\ RS\\ R\ R\ R44SjjrSrU=r$)Gemma4PatchEmbeduLinear patch embedding with a 2D position-embedding table. Unlike the standard ViT PatchEmbed (Conv2d), Gemma4 uses a Linear projection on flattened patches plus a separate 2D position-embedding table applied via one-hot lookup. Inputs may be supplied as: * ``(B, C, H, W)`` raw images — patchified inline in P-P-C flat order. * ``(B, N, P*P*C)`` pre-patchified P-P-C patches (NaFlex loader output). * ``(B, N, Ph, Pw, C)`` pre-patchified unflattened patches. * ``dict`` with keys ``patches`` / ``patch_coord`` / ``patch_valid``. All input-dispatch logic is owned by this module; the parent model simply calls ``self.patch_embed(x, patch_coord, patch_valid)`` and unpacks the returned tensors. N patch_sizein_chans embed_dimposition_embedding_sizer#cH>XVS.n[T U]5 [U5UlX0lX@lURup[ R"X(-U -U4SS0UD6Ul[ R"[R"SXC40UD65Ul UR5 g)Nr%r(FrX)r.r/rrrrr0r1 input_proj Parameterr4r5position_embedding_tabler6) r7rrrrr&r'r8phpwr9s r:r/Gemma4PatchEmbed.__init__s / #J/"'>$))HMB$6 TTQST(* KK2 D D) % r<c,[URSS9 g)N{Gz?std)rrrAs r:r6!Gemma4PatchEmbed.reset_parameterss66DAr< batch_sizepHpWr&c~[R"X$S9n[R"X4S9n[R"XVSS9upx[R"UR 5UR 5/SS9n U R S5R USS5n [R"XU-[RUS9n X4$)z:Row-major grid coords in external NaFlex ``(y, x)`` order.)r&ij)indexingrWrYrr) r4rmeshgridstackflattenrdronesrQ) r7rrrr&ysxsgrid_ygrid_x patch_coord patch_valids r:_default_patch_coord%Gemma4PatchEmbed._default_patch_coords\\" , \\" ,>kk6>>#3V^^5E"FBO !++A.55j"bI jj"WEJJvV ''r<rpadding_positionscBURSS9n[R"X0RS9nUR SSSS5R UR 5nX@R -nURSS9n[R"URS5S U5nU$) zCompute position embeddings via one-hot matmul against the 2D table. ``position_ids`` follows the Gemma4-internal ``(x, y)`` convention. rmin num_classesrXrrYrW) rFFone_hotrpermuterrsumr4whererd)r7rrclamped_positionsrposition_embeddingss r:_position_embeddings%Gemma4PatchEmbed._position_embeddingss)..1.5))-;W;WX//!Q1-001N1NO%(E(EE155!5<#kk  ' ' +S2E #"r<rCrrc[U[5(a)URSU5nURSU5nUSnURupEURS:XaFUR upgpUc$UR XhU-X-UR5up#[XU4SSS9upOURS:XaAURS S SS S 5RUR S UR S S 5nOvURS :XaMUR upjn XU--n URXjXEU 5RS S SS S 5RXjU 5nO[SURS35eUc [S5eUcfUS :HRS S9n U R5(aU )nO;[R "UR SS [R"URS9nS US- -nUR%UR'UR$R(R*55nUR-SS9nU)nXR/X5-nXU4$)aNormalize inputs, patchify / project / position-embed. Accepts external NaFlex ``patch_coord`` (y, x) / ``patch_valid`` inputs and returns the internal (x, y) form used by the rest of the model. Returns: embeddings: (B, N, embed_dim) patch embeddings with position info added. position_ids: (B, N, 2) Gemma4-internal (x, y) coords. padding_positions: (B, N) True for padding tokens. rrpatchesNF)pad channels_lastrrrXrrWz&Expected input ndim in (3, 4, 5); got .z1patch_coord is required for pre-patchified input.rYr?)rW)dims) isinstancedictgetrrfr[rr&rrreshapeview ValueErrorallanyr4rrQrrweightr'flipr)r7rCrrrrB_HWNPPCCsentinelrrs r:rHGemma4PatchEmbed.forwards8 a  %% {;K%% {;K) A 66Q;JA!"+/+D+DQRQRQXZ[ZbZb+c( !!"X5NDAq VVq[ !Q1a(00QWWQZLA VVq[IA#R AqRQ'//1aA>FFqSQA8B   PQ Q  #r)..2.6H||~~'i #jj%%bq)KDVDV SM OOADD!7!7!=!=> ?#''U'3 (L )),J J 111r<)rrrrr)r(NNrJNN)rKrLrMrNrOr rPr r/r6r4r&rRrrrstrrrHrSrTrUs@r:rrsj$79 +0 c5c?23    &)     .B((( ( LL ( u||U\\) * ( #,,# %||#  #.3726 @2U\\4U\\(9#::;@2"%,,/@2"%,,/ @2 u||U\\5<<7 8 @2@2r<rc8^\rSrSr%Sr\R R\\ S'SS\ S\ S\ S\ \ S \ S \ S \ S \S S4U4Sjjjr SS\RS\RS\RS\ \RS \R4 SjjrSrU=r$)Gemma4Attentioni\aGemma4 Vision Attention with QKV normalization and 2D RoPE. Key features: - Separate Q, K, V projections (not fused) - RMSNorm on Q, K (with scale) and V (without scale) - 2D RoPE applied after normalization - Attention scale = 1.0 (since QK are normalized) fused_attnNrZ num_headsrv num_kv_heads attn_drop proj_dropnorm_epsuse_clipped_linearsr#c b>XS.n [T U]5 X lX0lU=(d UUlX R-Ul[ 5Ul[XU-4SU0U D6Ul [XRU-4SU0U D6Ul [XRU-4SU0U D6Ul [X#-U4SU0U D6Ul [U4USS.U D6Ul[U4USS.U D6Ul[U4USS.U D6Ul["R$"U5Ul["R$"U5Ulg)Nr%r"TepsaffineF)r.r/rrvr num_kv_groupsrrrq_projk_projv_projo_projrq_normk_normv_normr0Dropoutrr) r7rZrrvrrrrrr&r'r8r9s r:r/Gemma4Attention.__init__hs2 / " (5I&*;*;;(*+CX1EmSfmjlm +C1B1BX1Mu[nurtu +C1B1BX1Mu[nurtu +I,@#mSfmjlm hHHTHRH hHHTHRH hIHUIbI I.I.r<rCrope_cosrope_sin attn_maskc URupVnURU5RXVURUR5nUR U5RXVUR UR5n URU5RXVUR UR5n URU5nURU 5n URU 5n [XUSSS9n[XUSSS9n URSS5nU RSS5n U RSS5n URS:a4U RURSS9n U RURSS9n UR(aA[ R""UU U UUR$(aUR&R(OSSS9nO?XRSS 5-n UbX-n U R+S S9n UR'U 5n X-nURSS5R-XVS 5nUR/U5nUR1U5nU$) NrX)rfrbrrYrr)r  dropout_pscalerW)r[rrrrvrrrrrrrrrrrepeat_interleaverrscaled_dot_product_attentiontrainingrpsoftmaxrrr) r7rCr r r rrrqrpvattns r:rHGemma4Attention.forwards''a KKN  dnndmm D KKN  d&7&7 G KKN  d&7&7 G KKN KKN KKN (XAUV W 'XAUV W KK1  KK1  KK1     !##D$6$6A#>A##D$6$6A#>A ??..#.2mm$..** A{{2r**D$'<>$'DA KK1  % %aB / KKN NN1 r<)rrrvrrrrrrrrrrr) @Nrrư>FNNrE)rKrLrMrNrOr4jitFinalrQ__annotations__rPrr@r/rRrHrSrTrUs@r:rr\s %%  *."""(- / / / / #3- /  / / /"& /  / /N15 5||5ll5ll 5   - 5  55r<rc ^\rSrSrSrSS\S\S\\S\S\ S S4 U4S jjjr S \ RS \ R4S jr S rU=r$)Gemma4GatedMlpizyGated MLP for Gemma4 Vision Encoder. Uses GELUTanh activation: output = down_proj(gelu_tanh(gate_proj(x)) * up_proj(x)) Nr hidden_features act_layerdroprr#c>XgS.n[T U]5 [X4SU0UD6Ul[X4SU0UD6Ul[X!4SU0UD6UlUbU"5O[ R"SS9Ul[ R"U5Ul g)Nr%r"tanh approximate) r.r/r gate_projup_proj down_projr0GELUactr r%) r7r r#r$r%rr&r'r8r9s r:r/Gemma4GatedMlp.__init__s / .{sYlsprs,[qWjqnpq .sYlsprs"+"79;RWWQW=XJJt$ r<rCc URURURURU55UR U5-55$rE)r%r,r.r*r+rGs r:rHGemma4GatedMlp.forwards:yy1B(CdllSTo(UVWWr<)r.r,r%r*r+)NrFNN)rKrLrMrNrOrPrrr@rQr/r4rRrHrSrTrUs@r:r"r"s-1(-%%!% ) %  % "& % %%$XX%,,XXr<r"c^\rSrSrSrSS\S\S\S\S\\S \S \S \S \S \\S\ SS4U4Sjjjr SS\ RS\ RS\ RS\\ RS\ R4 Sjjr SrU=r$) Gemma4BlockizGemma4 Vision Encoder Block with 4-norm sandwich pattern. Unlike standard ViT (pre-norm with 2 norms), Gemma4 uses: - input_layernorm (norm1) + post_attention_layernorm (norm2) - pre_feedforward_layernorm (norm3) + post_feedforward_layernorm (norm4) NrZrrvintermediate_sizerrrr drop_pathr$rr#c >XS.n[TU]5 [U4SU0UD6Ul[ SUUUUUUUU S.UD6Ul[U4SU0UD6Ul[U4SU0UD6Ul[SUUU U S.UD6Ul [U4SU0UD6Ul U S:a[U 5Ulg[R"5Ulg)Nr%r)rZrrvrrrrr)r r#r$rrr-)r.r/rnorm1rrnorm2norm3r"mlpnorm4rr0Identityr5)r7rZrrvr4rrrrr5r$rr&r'r8r9s r:r/Gemma4Block.__init__s  / S5h5"5 #  % 3     S5h5"5 S5h5"5 ! - 3    S5h5"5 09C),R[[]r<rCr r r c$UnURU5nURXX4S9nURU5nXPRU5-nUnUR U5nUR U5nUR U5nXPRU5-nU$)Nr )r7rr8r5r9r:r;)r7rCr r r residuals r:rHGemma4Block.forwards JJqM IIa8I A JJqM ~~a( ( JJqM HHQK JJqM ~~a( (r<)rr5r:r7r8r9r;) NrrrrNFNNrE)rKrLrMrNrOrPrr@rrQr/r4rRrHrSrTrUs@r:r3r3s+/"""",0(-(S(S(S (S # (S #3- (S(S(S(S(S )(S"&(S (S(S^15 ||llll    -   r<r3c 4^\rSrSrSrS S\S\4U4SjjjrS\RS\RS\ \R\R44S jr S\RS\RS \RS\ \R\R44S jr S r U=r $)Gemma4VisionPooleri'uSpatial pooling for Gemma4 vision encoder output. Pools patches by averaging within k×k grid cells based on position coordinates. Output is scaled by sqrt(hidden_size). hidden_sizepooling_kernel_sizecN>[TU]5 XlUS-UlX lg)Nr)r.r/rDroot_hidden_sizerE)r7rDrEr9s r:r/Gemma4VisionPooler.__init__.s' & +S 0#6 r< hidden_statesrr#c TURSnURnXD-nX5-S:wa[SUSUSUS35eX5-nURSS9nUSR S S S 9SS-n[ R "XtS S 9n U SX-U S--n [R"U R5U5R5U- n U RSS5UR5-n [ R"U S:HRSS95n U RUR5U 4$)u 2D spatial pooling on a ``k × k`` grid (k = ``self.pooling_kernel_size``). ``N`` patches are binned into ``k^2``-sized cells, so the pool requires ``N % k^2 == 0`` (caller ensures both grid dims divide by k upstream). ``position_ids`` follows the Gemma4-internal ``(x, y)`` convention. rrz Cannot pool z tokens with k=z: N must be divisible by k^2=z.. Both grid dimensions must be divisible by k.r.rrWTrZkeepdimfloor) rounding_mode.rrXrY)r[rErrFmaxr4divrrlongr@r logical_notrrr') r7rIrrrp k_squared output_lengthrmax_x kernel_idxsweightsoutputmasks r:_avg_pool_by_positions)Gemma4VisionPooler._avg_pool_by_positions4sO    "  $ $E =A qc3PQZP[\?@  (..1.5!&)--"d-CAFJii 1GL !&)UZ;v;N,NN ))K,,. >DDFR""1a(=+>+>+@@  'Q,!3!3!3!:;yy,,-t33r<rcURURS5S5nURX5upXR-nX4$)uGSpatial pool with ``pooling_kernel_size × pooling_kernel_size`` cells. Args: hidden_states: (B, N, D) encoder output. position_ids: (B, N, 2) Gemma4-internal ``(x, y)`` coords. padding_positions: (B, N) True for padding tokens. Returns: pooled hidden states (B, N // k^2, D) and validity mask (B, N // k^2). rWr) masked_fillrdr\rG)r7rIrr pooler_masks r:rHGemma4VisionPooler.forwardSsM"&112C2M2Mb2QSVW %)%@%@%]" %(=(== ))r<)rDrErG)r)rKrLrMrNrOrPr/r4rRr r\rHrSrTrUs@r:rCrC's 7C7c77 4 <<4 ,,4 u||U\\) * 4>* <<* ,,* %|| * u||U\\) * **r<rCc-P^\rSrSrSrS=S\\\\\44S\S\S\S\S \S \S \S \ \S \S\ S\ S\S\S\ S\ S\ S\ S\ S\ \ S\SS4,U4Sjjjr \RR S>S\S\ SS4Sjj5r\RR S\\4Sj5r\RR S\\\44Sj5r\RR S?S \ S\\\44S!jj5r\RR S@S"\ SS4S#jj5r\RR S@S$\ SS4S%jj5rS&\R4SS4S'jrSAS&\R4S(\R4S)\R4S*\ \ \\R4/S4S+\ \S\R44 S,jjrSAS&\\R4\\\R444S-\ \R4S.\ \R4S\R44S/jjrS?S&\R4S0\ S\R44S1jjrSAS&\\R4\\\R444S-\ \R4S.\ \R4S\R44S2jjrSBS&\\R4\\\R444S-\ \R4S.\ \R4S3\ \\\ \4S4\ S5\ S6\S7\ S\\ \R4\\R4\ \R4444S8jjr!SCS3\\\ \4S9\ S:\ S\ \4S;jjr"SUUS.n[TU]5 US;d SU<35eX@lU=Ul=UlUlSUlSUlSUlSUl [U5Ul Xl UUl U=(d [[R SS9n[#SURUUU S .UD6Ul['SUU S .UD6Ul[*R,"SUU5Vs/sHnUR/5PM nn[R0"[3U5Vs/sHnUUUU U U UUUUUUS . UD6PM sn5UlUS 9UlU(aOUR=S [*R>"U40UD65 UR=S[*R>"U40UD65 OSUl SUl![EUR5n[3U5Vs/sHnUUSU3S9PM snUl$US:XaSOUUl%US:waURMSS9 ggs snfs snfs snf)Nr%)softavgnonez>global_pool must be one of 'soft', 'avg', 'none' (or ''); got rNLCFr'r()rrrrr) rZrrvrr4rrrr5r$r)rDrEstd_bias std_scaleblocks.)num_chs reductionmoduleskipreset needs_resetr-)'r.r/rd num_featureshead_hidden_sizerr output_fmtnum_prefix_tokensgrad_checkpointingrrrErrr0r-r patch_embedrt rotary_embr4linspaceitem ModuleListrir3blocksrCpoolerr3r5rqrrrQr feature_infoweight_init_mode init_weights)r7rcrrrdrrerrvrr4rrwrrErfrrgrhrir$rjr&r'r8rCdprr_redr9s r:r/Gemma4VitEncoder.__init__|s{4 / 99 ]L[O \ ]9&ENNND1DN !""'$J/#6 #6 Ef!E , $;    2 !  "'>5!IJ!IAqvvx!IJmm5\% "  #!)"3!((a&#$7  "%   &)! 3     U[[-Ib-I J  ekk).Jr.J K DM!DN 4??#dijodpqdp_`T)tgVWUXMZdpq+6&+@k &   %  0 !WK% Lrs'H+ $H0%H5moderzc<U=(d URnUS;deURb)[RR UR5 UR b)[RR UR 5 [[XS9U5 g)aoInitialize model weights. Args: mode: Init mode. '' applies trunc-normal-TF Linear init; 'reset' only calls ``reset_parameters`` on each sub-module. needs_reset: If True, call ``reset_parameters`` on modules that have one (for post-``to_empty()`` reinit). Set to False during ``__init__`` since modules already self-initialize there. )rorxNry) rrqr0initzeros_rrones_rget_init_weights_gemma4_vitr7rrzs r:rGemma4VitEncoder.init_weightssq,t,,}$$$ == $ GGNN4== ) >> % GGMM$.. )/NPTUr<cS1$)Nz$patch_embed.position_embedding_tabler-rAs r:no_weight_decay Gemma4VitEncoder.no_weight_decays 677r<cUR$zHReturn the 2-tuple patch size. For NaFlex dataloader / transform wiring.)rrAs r:get_patch_sizeGemma4VitEncoder.get_patch_sizesr<coarsec[SSS/S9$)Nz^patch_embed|^rotary_emb)z^blocks\.(\d+)N)z ^pooler|^std_)istemrrr7rs r: group_matcherGemma4VitEncoder.group_matcher s,-/KL  r<enablecXlgrE)rr7rs r:set_grad_checkpointing'Gemma4VitEncoder.set_grad_checkpointings"(r<enabledclUR5H n[U[5(dMXlM" g)aoToggle the ``Gemma4ClippableLinear`` clamp ops. Gemma4's pretrained E4B checkpoint ships finite clamp buffers on every projection, which can saturate and stall gradient flow during classifier fine-tuning. Setting ``enabled=False`` skips both the input and output clamps in the forward pass (buffers are left untouched so the call is reversible and checkpoint-safe). N)modulesrrr")r7rmods r:set_clamp_enabled"Gemma4VitEncoder.set_clamp_enableds'<<>C#455")"r<rCcURS:wdURS:wagURSSup#URupEURnXF-XV-pX'-S:wdX8-S:wa[ SUSUSUSUS USUS 3 5eg) zWhen using the soft-token pooler, raw-image H/W must divide by ``patch_size * pooling_kernel_size`` so the pool cell grid is integral. Pre-patchified / NaFlex inputs are assumed to be conformant already. rrlNrrz Image size (z, z=) must be divisible by (patch_size * pooling_kernel_size) = (z3) when global_pool='soft'. Resize to multiples of (z#), or use global_pool='avg'/'none'.)rfrdr[rrEr) r7rCrrrrrpcell_hcell_ws r:_assert_raw_img_conformant+Gemma4VitEncoder._assert_raw_img_conformant$s 66Q;$**f4 wwrs|  $ $ :?ajAoqcA3'99?6(K++1("VH<_a .r<rrblock_callbackmax_block_indexc 8URSSupgURX5upSn UR5(aR[R"USSXqR UR S9n U RUSS2SSSS24[S55 Uc URn OURSUS-n [U 5H[upUR(a1[RR5(d[XXU 5nOU "XXS9nUcMSU"X5 M] U$)z?RoPE + transformer-block pipeline over already-embedded tokens.NrXrr%z-infr?)r[rrr4zerosr&r' masked_fill_r@r enumeraterr is_scriptingr)r7rCrrrrrrr r r rrblks r:_encodeGemma4VitEncoder._encode6swwr{!__Q=,0  " " Aq!QxxqwwOI  " "#4QdA5E#Ff V  "[[F[[!5/A"56F'FA&&uyy/E/E/G/Gsx9EXC)q$ (r<rrcUR[U[5(dUOUS5 URXU5upnUR XU5$)z9Raw patch tokens pre-pool. Returns ``(B, N, embed_dim)``.r)rrrrr)r7rCrrrrs r:forward_features!Gemma4VitEncoder.forward_featuresWsN ''At1D1D!I,W-1-=-=ak-Z**||A->??r< pre_logitsc[S5e)Nz;Gemma4VitEncoder does not support classification use cases.)NotImplementedError)r7rCrs r: forward_headGemma4VitEncoder.forward_headbs!"_``r<cPUR[U[5(dUOUS5 URXU5upnUR XU5nUR S:Xa>UR XU5upURbXR- UR-nU$UR S:XauUR5(aQURURS5S5nURSS9U)RSSS 9RSS 9- nU$URSS9nU$) uEncode + apply the configured pool. Output shape depends on ``self.global_pool``: ``'soft'`` → ``(B, num_soft_tokens, D)`` ``'avg'`` → ``(B, D)`` ``'none'`` → ``(B, N, D)`` (raw patch tokens, identical to forward_features) rrlrmrWrrrYTrLr)rrrrrrdrrqrrrr_rdrrFmean)r7rCrrrrrs r:rHGemma4VitEncoder.forwardes% ''At1D1D!I,W-1-=-=ak-Z** LL*; <   v %;;q0ABDA}}(&$..8   & $$&&MM"3"="=b"A3GEEaEL%6$6#;#;4#;#P#V#V[\#V#]] FFqFMr<indicesnorm stop_earlyr}intermediates_onlyc x^^US;deUS:Hn [[UR5U5umn [U[5(dUOUSn U R n UR U 5 URXU5upn/mS[S[RSS4UU4SjjnSnU(a%[RR5(dU nURUU UUUS 9nU (aU S :wa [S 5eU RS n[U S R!5R#55S-n[U SR!5R#55S-nTVs/sH7nUR%UUUS5R'S SSS5R)5PM9 snmU(aT$UT4$s snf)aIForward features returning intermediates. Args: x: Input tensor ``(B, C, H, W)`` or NaFlex pre-patchified tensor/dict. patch_coord: ``(B, N, 2)`` patch coords in NaFlex external (y, x) convention. patch_valid: ``(B, N)`` boolean mask, True for valid tokens. indices: Block indices to return intermediates for. norm: Not used (no final norm in Gemma4 encoder). stop_early: Stop iterating after last needed intermediate. output_fmt: Output format ('NCHW' or 'NLC'). NCHW requires a fixed full grid. intermediates_only: Only return intermediate features. )NCHWrprrryr#Nc6>UT;aTRU5 ggrE)r)rr intermediates take_indicess r:_cb3Gemma4VitEncoder.forward_intermediates.._cbsL $$Q'!r<)rrrz:output_fmt='NCHW' requires a raw image (B, C, H, W) input.rrKrrPrWrrX)rlenrrrrfrrrPr4rRrrrrr[rQrrr contiguous)r7rCrrrrrr}rr max_indexrawraw_input_ndimrrrrrrrrrrs @@r:forward_intermediates&Gemma4VitEncoder.forward_intermediatess._,,,&"6s4;;7G"Q i!!T**a)  '',-1-=-=ak-Z**,.  (3 (5<< (D ( ( eii4466'O LL  +   " !]^^""1%A\&)--/4467!;B\&)--/4467!;B`mn`m[\QYYq"b"5==aAqITTV`mnM  - os)>F7 prune_norm prune_headcv[[UR5U5upEURSUS-UlU$)zPrune layers not required for specified intermediates. ``prune_head`` is accepted for API compatibility; the encoder has no classifier head so it's a no-op here. (The wrapping ``Gemma4VitClassifier`` handles its own head pruning.) Nr)rrr)r7rrrrrs r:prune_intermediate_layers*Gemma4VitEncoder.prune_intermediate_layerss8#7s4;;7G"Q kk.9q=1 r<)rrrrdrr|rr{r~r}rrrrErrqrrrr)rrrrlrrrrN rrrrFFrrrNroNNroTFTrNNNFFrFrFT)%rKrLrMrNrOr rPr rrr@rQrr/r4rignorerr rrrrrrrrRrrrrrHrrrrSrTrUs@r:rrjs/&58 % *.%)" %+0'( %(-$'$'$',0!1m1CsCx01m1m1 m1  m1  m1m1m1m1#3-m1 #m1m1m1&)m1"%m1 !m1""&#m1$"%m1&"'m1(")m1* )+m1,-m12 3m1m1^ YYVVVVV, YY8S88 YYc3h YY D T#s(^   YY)T)T)) YY * * * *ELLT.MQ-1 || ,, %||  %XsELL.A4.G%HI  &c]  H3726 @U\\4U\\(9#::; @"%,,/ @"%,,/ @  @aellaaa 3726 U\\4U\\(9#::; "%,,/ "%,,/   J37267;$$',> U\\4U\\(9#::;> "%,,/> "%,,/ > eCcN34 >  > > > !%>  tELL!5tELL7I)I#JJ K> D./$# 3S >*  c r<c50^\rSrSrSrS=S\\\\\44S\S\S\S\S \S \S \S \S \S\ \S\S\ S\ S\S\S\ S\ S\ S\ S\ S\ S\ S\ \ S\SS44U4Sjjjr \RR S>S\S \ SS4S!jj5r\RR S\\4S"j5r\RR S?S#\ S\\\44S$jj5r\RR S@S%\ SS4S&jj5r\RR S@S'\ SS4S(jj5r\RR S\\\44S)j5r\RR S\R64S*j5rSAS\S\ \SS4S+jjrSBS,\\R<\\\R<44S-\ \R<S.\ \R<S\R<4S/jjrSCS,\R<S.\ \R<S0\ S\R<4S1jjr SBS,\\R<\\\R<44S-\ \R<S.\ \R<S\R<4S2jjr!SDS,\\R<\\\R<44S-\ \R<S.\ \R<S3\ \\\"\4S4\ S5\ S6\S7\ S\\"\R<\\R<\"\R<444S8jjr#SES3\\\"\4S9\ S:\ S\"\4S;jjr$SUUS.n[TU]5 US;d SU<S35eUS;d SU<S35eX@lXPlX`l[ S!0SU_S U_S U_S U_S U_S U_SU _SU _SU _SU _SU _SU_SU_SU_SU_SU_SU_SU_SU_SU_SU_UD6UlU(a[U4U SS.UD6O[R"5Ul [R"U5Ul US:a[R"Xt40UD6O[R"5UlUR R=UlUlUR R"UlUR R$UlUR R&UlUR R(UlUS :wav[+UR[R5(aLURR,b4[R.R1URR,5 gggg)"Nr%rmrnroBGemma4VitClassifier global_pool must be 'avg', 'none', or '' (got z>); use Gemma4VitEncoder directly for 'soft' VLM-style pooling.)rornrlzDGemma4VitClassifier encoder_pool must be '', 'none', or 'soft' (got z).rcrrrdrrerrvrr4rrwrrErfrrgrhrir$rjFrrrwr-)r.r/rrdrrencoderrr0r<rr  head_dropr1headr{r|rrrrrr(rr)r7rcrrrrdrrrerrvrr4rrwrrErfrrrrgrhrir$rjr&r'r8r9s r:r/Gemma4VitClassifier.__init__su< / 11 KPQ\P_`J K K133 fRS_Rbbd e f3&&((  !  %         & 0  " %< !4 $ !4! "*# $*% &*' ( ) *$-  0MWGIH8EHRH\^\g\g\i I.?JQBIIi;;TVT_T_Ta 59LL4M4MMD1//,,11 LL55 $ = = & $))RYY//DIINN4Ntyy~~.5O/ !r<rrzc URRXS9 [UR[R 5(aLURR b4[RRURR 5 ggg)z>Init encoder + classifier-specific pieces (head bias to zero).rrzN) rrrrr0r1r(rrrs r:r Gemma4VitClassifier.init_weights@s\ !!t!E dii + + 0J GGNN499>> *1K +r<chURR5Vs1sHnSU3iM sn$s snf)Nencoder.)rr)r7rps r:r#Gemma4VitClassifier.no_weight_decayGs/(, (D(D(FG(F1(1#(FGGGs/rc[SS/S9$)Nz*^encoder\.patch_embed|^encoder\.rotary_emb)z^encoder\.blocks\.(\d+)Nrrrs r:r!Gemma4VitClassifier.group_matcherKs>67  r<rc:URRU5 grE)rrrs r:r*Gemma4VitClassifier.set_grad_checkpointingSs ++F3r<rc:URRU5 grE)rr)r7rs r:r%Gemma4VitClassifier.set_clamp_enabledWs &&w/r<c6URR5$r)rrrAs r:r"Gemma4VitClassifier.get_patch_size[s||**,,r<cUR$rE)rrAs r:get_classifier"Gemma4VitClassifier.get_classifier`s yyr<cXlUbUS;d SU<S35eX lUS:a![R"URU5O[R "5Ul[UR [R5(aLUR Rb4[RRUR R5 ggg)Nrr)r) rrdr0r1rr<rrr(rr)r7rrds r:reset_classifier$Gemma4VitClassifier.reset_classifierds&  ""55 fTU`Tccde f5* >IAoBIIdnnk:SUS^S^S` dii + + 0J GGNN499>> *1K +r<rCrrcvURS:XaURXUS9$URRXUS9$)u-Pre-head features. Shape depends on ``encoder_pool``: ``''`` / ``'none'``: raw patch tokens ``(B, N, embed_dim)``. ``'soft'``: spatial ``k×k`` soft-token pool output ``(B, num_soft_tokens, embed_dim)`` (with ``√D`` scale + optional ``std_bias/std_scale`` baked in by the encoder). rlrr)rrr)r7rCrrs r:r$Gemma4VitClassifier.forward_featuresns@    &<< <T T||,,QU`,aar<rc~URS:XarURS:XdUcURSS9nOOURU)R S5S5nUR SS9UR SSS9R SS 9- nURU5nURU5nU(aU$URU5$) uPool (if configured) → norm → head_drop → head. Args: x: pre-head features from ``forward_features``. patch_valid: ``(B, N)`` valid mask for masked mean over raw patch tokens. Ignored when ``encoder_pool='soft'`` (the soft pooler has already collapsed padding tokens, and the new token count no longer aligns with ``patch_valid``). pre_logits: If True, return pre-classifier features. rmrlrrYrWrTrLr) rdrrr_rdrrFrrr)r7rCrrs r:r Gemma4VitClassifier.forward_heads   u $  F*k.AFFqFMMMK<":":2">DEEaEL;??q$?#G#M#MRS#M#TT IIaL NN1 q0DIIaL0r<c[U[5(a)URSU5nURSU5nUSnUc-Ub*US:HRSS9nUR 5(aU)nUR XUS9nUR XSS9$)NrrrrWrYr )r)rrrrrrr)r7rCrrrfeatss r:rHGemma4VitClassifier.forwards a  %% {;K%% {;K) A  ;#:#r)..2.6H||~~'i %%ak%Z   @@r<rrrr}rc BURRUUUUUUUUS9$)N)rrrrrr}r)rr) r7rCrrrrrr}rs r:r)Gemma4VitClassifier.forward_intermediatess8||11 ##!!12  r<rrcjURRXSS9nU(aURS5 U$)NF)rrr)rrr)r7rrrrs r:r-Gemma4VitClassifier.prune_intermediate_layerss5 ||==gin=o   ! !! $r<) rrrrrdrrr|rrr{rr)rrrirmrorrrrNrrrrrFFTrrrrNroNNrrrrEr)NFrr)'rKrLrMrNrOr rPr rrr@rQrr/r4rrrr rrrrrrrr0ModulerrrRrrrHrrrrSrTrUs@r:rrsA,58 #$ " *.%)" %+0'( %(-#"$'$'$',0!9Q/CsCx01Q/Q/ Q/  Q/  Q/Q/Q/Q/Q/Q/#3-Q/ #Q/Q/Q/ &)!Q/""%#Q/$%Q/&"&'Q/()Q/*+Q/,"-Q/."/Q/0"1Q/2 )3Q/45Q/: ;Q/Q/f YY+++++  YYHSHH YY D T#s(^   YY4T4T44 YY0000 YY-c3h-- YY +C+hsm+W[+3726 bU\\4U\\(9#::;b"%,,/b"%,,/ b  b*37$ 1||1"%,,/1 1  1<3726 AU\\4U\\(9#::;A"%,,/A"%,,/ A  A837267;$$', U\\4U\\(9#::; "%,,/ "%,,/  eCcN34      !%  tELL!5tELL7I)I#JJ K 0./$# 3S >*    c  r<Trvnamerzcr[U[R5(aL[URSS9 UR b*[R RUR 5 gg[US5(aUR5 gU(a#[US5(aUR5 ggg)zPer-module init for Gemma4Vit (trunc-normal-TF for Linear weights). Args: module: Module to initialize. name: Dotted module name (from ``named_apply``). needs_reset: If True, call ``reset_parameters`` on modules that define one. rrNrr6) rr0r1rrr(rrhasattrrr6)rvrrzs r:init_weights_gemma4_vitrs&"))$$D1 ;; " GGNN6;; ' #  ( ( );<<!=r<rc[[US9$)Nry)rrrs r:rrs * DDr< state_dictmodelc0nSnUR5HupESnUHnURU5(dMUn O UcURS5(aXRU'MIU[U5SnSU;aM_URSS5nURSS5nURS S 5nURS S 5nURS S5nURSS5nURSS5nXRU'M U$)uConvert HuggingFace Gemma4 vision encoder weights → ``Gemma4VitEncoder`` keys. Pure key remapping — no value transforms. The encoder consumes patches in HF's native C-P-P flat layout (via ``batch_patchify(channels_last=False)`` in ``Gemma4PatchEmbed``), so ``input_proj.weight`` passes through unchanged. ``Gemma4ClippableLinear`` preserves HF's ``.linear.weight`` + clamp-buffer structure, so those pass through too. )zmodel.vision_tower.zmodel.vision_model.z vision_model.z vision_tower.N) patch_embed.rsstd_zpooler.z rotary_emb.rzpatch_embedder.rzencoder.layers.rsz.input_layernorm.z.norm1.z.post_attention_layernorm.z.norm2.z.pre_feedforward_layernorm.z.norm3.z.post_feedforward_layernorm.z.norm4.z .self_attn.z.attn.)items startswithrreplace) rrout_dict hf_prefixesrprmatched_prefixprefixnew_ks r:checkpoint_filter_fn_encoderr'sHbK  "!F||F##!'"  !||YZZ #n%&' 5  /@ /; 19= :IF ;YG #&"!  .' ( ) r<z"timm/gemma4_vit_167m.gemma4_e4b_itzencoder.patch_embed.input_projz apache-2.0) hf_hub_idrJlicensez"timm/gemma4_vit_570m.gemma4_31b_itz&timm/gemma4_vit_167m_enc.gemma4_e4b_it)rMrNz&timm/gemma4_vit_570m_enc.gemma4_31b_it)zgemma4_vit_167m.gemma4_e4b_itzgemma4_vit_570m.gemma4_31b_itz!gemma4_vit_167m_enc.gemma4_e4b_itz!gemma4_vit_570m_enc.gemma4_31b_itrrrrr)rrerrvr4rfriHi)rrerrvr4rfc L[[SS9n[SSU0[U40UD6D6$)zGemma4 ~167M (E2B/E4B vision tower) classifier. Masked mean pool over patch tokens + norm + linear classifier. Output: ``(B, num_classes)``. For the native VLM encoder interface (soft-token output), use ``gemma4_vit_167m_enc``. Trr1)gemma4_vit_167m)r _167M_ARCHr?r1r< model_argss r:rSrSs0jT2J ( pz pUYZdUohnUo ppr<c L[[SS9n[SSU0[U40UD6D6$)uGemma4 ~167M (E2B/E4B vision tower) — native VLM encoder. ``global_pool='soft'`` applies the spatial ``k×k`` soft-token pool + √D scale; output: ``(B, num_soft_tokens, embed_dim)``. Bit-perfect with HF ``Gemma4VisionModel`` on matching weights. rlrdr1)gemma4_vit_167m_enc)rrTr=rUs r:rYrY0jf5J % q qVZ[eVpioVp qqr<c L[[SS9n[SSU0[U40UD6D6$)aSGemma4 ~570M (26B/31B vision tower) classifier. Masked mean pool over patch tokens + norm + linear classifier. Output: ``(B, num_classes)``. The classifier uses ``encoder.forward_features`` and does its own pool + norm; the encoder's ``std_bias/std_scale`` (which only applies in ``'soft'`` pool mode) is not used on the classifier path. TrRr1)gemma4_vit_570m)r _570M_ARCHr?rUs r:r\r\s0jT2J ( pz pUYZdUohnUo ppr<c L[[SS9n[SSU0[U40UD6D6$)uGemma4 ~570M (26B/31B vision tower) — native VLM encoder. ``global_pool='soft'`` output ``(B, num_soft_tokens, embed_dim)`` with ``std_bias/std_scale`` standardization applied post-pool (HF-native ordering — bit-perfect with ``Gemma4VisionModel``). rlrXr1)gemma4_vit_570m_enc)rr]r=rUs r:r_r_rZr<)rX)rXrXrr)ro)IrOmath functoolsrtypingrrrrrr r r r4torch.nnr0torch.nn.functional functionalr timm.datar r timm.layersrrrrr_builderr _featuresr _manipulaterr _registryrr naflexvitr__all__rrrRr_rPrerrrtrrr"r3rCrrrrQrrr'r/r=r?rL default_cfgsrrTr]rSrYr\r_r-r<r:ros  III AVV*+0<% 4 5*BII*Z(5<<(ELL( . << . \\ .\\ . .  \\ .  & <<& \\&\\& &  &  \\ &4MbiiM`O2ryyO2dcbiicLXRYYX8F"))FR@*@*Fkryyk\ {")){|"BII"S"D"\`"$EcETEXE ,ell*+,, #u|| ,^ell*+$ #u|| 0   Td  3 D Wj cT#s(^2%%)63& &*63& *.:**.:*#& 0       qq;NqqrDr?Orr q q;N q qrDr?Orrr<