3jufdSSKJr SSKrSSKJrJrJr SSKrSSKJ s J r SSKJ r SSK J r SSKJr SSKJrJrJrJr S!Sjr"S S \ R,5r"S S \ R,5r"S S\ R25r"SS\ R25r"SS\ R25r"SS\ R,5r"SS\ R25r"SS\ R25r"SS\ R25r "SS\ R25r!S"Sjr"S#S$Sjjr#S#S$Sjjr$S#S$S jjr%g)%) annotationsN)AnyOptionalSequence)nn) checkpoint) KORNIA_CHECK)DropPath LayerNorm2dwindow_partitionwindow_unpartitionc6[U[5(aX4$U$N) isinstanceint)xs P/home/wildlama/miniconda3/lib/python3.13/site-packages/kornia/models/tiny_vit.py _make_pairr#s3''A6.Q.cn^\rSrSrSrSSS\R 4SU4SjjjrSrU=r $)ConvBN'aImplement a sequential block containing a convolution followed by Batch Normalization. Args: in_channels: The number of input channels. out_channels: The number of output channels. kernel_size: The size of the convolving kernel. Default: 1. stride: The stride of the convolution. Default: 1. padding: The zero-padding added to both sides of the input. Default: 0. groups: The number of blocked connections from input to output. Default: 1. rc >[TU]5 [R"XX4XVSS9Ul[R "U5UlU"5Ulg)NF)groupsbias)super__init__rConv2dc BatchNorm2dbnact) self in_channels out_channels kernel_sizestridepaddingr activation __class__s rrConvBN.__init__3s@ ;k7hmn...<r)r#r"r )r%rr&rr'rr(rr)rrrr*type[nn.Module]returnNone) __name__ __module__ __qualname____firstlineno____doc__rIdentityr__static_attributes__ __classcell__r+s@rrr'sr  &(kk              $      rrcJ^\rSrSrSr\R 4SU4SjjjrSrU=r $) PatchEmbedCaXPerform patch embedding using a series of convolutions. This module divides the input image into patches and projects them into the embedding dimension. Args: in_channels: The number of input image channels. embed_dim: The dimension of the resulting embeddings. activation: The activation layer to use. Default: :class:`nn.GELU`. c >[TU]5 [R"[ XS-SSS5U"5[ US-USSS55Ulg)Nr)rrr Sequentialrseq)r$r% embed_dimr*r+s rrPatchEmbed.__init__OsL == ;Q1a 8*,y\]~_hjkmnpqHr r)r@)r%rrArr*r-r.r/ r0r1r2r3r4rGELUrr6r7r8s@rr:r:Cs XZW^W^  rr:cl^\rSrSrSr\R S4SU4SjjjrSSjrSr U=r $) MBConvVazImplement the Mobile Inverted Residual Bottleneck Convolution layer. Args: in_channels: The number of input channels. out_channels: The number of output channels. expansion: The expansion ratio for the hidden dimension. kernel_size: The convolution kernel size. stride: The stride of the convolution. dropout: The dropout rate for the stochastic depth. c >[TU]5 [X-5n[XSUS9Ul[XfSSSXd5Ul[XbS5Ul[U5UlU"5Ul g)Nrr*r>) rrrrconv1conv2conv3r drop_pathr#)r$r%r&expansion_ratior*rNhidden_channelsr+s rrMBConv.__init__bsd k;<K! S OaAc O1= !),<rc URXRURURUR U5555-5$r)r#rNrMrLrKr$rs rforwardMBConv.forwardrs7xxNN4::djjA6O+PQQRRr)r#rKrLrMrN) r%rr&rrOfloatr*r-rNrVr.r/r torch.Tensorr.rX r0r1r2r3r4rrDrrTr6r7r8s@rrFrFVsd  ')gg      $       SSrrFcj^\rSrSrSr\R 4SU4SjjjrSSjrSr U=r $) PatchMergingvaImplement the patch merging layer for downsampling in TinyViT. Args: input_resolution: The height and width of the input feature map. dim: The number of input channels. out_dim: The number of output channels. act: The activation function class. Default: nn.GELU. c >[US;S5 [TU] 5 [U5Ul[ X#SUS9Ul[ X3SUSX5S9Ul[ X3S5Ulg)N)rr=zstride must be either 1 or 2rrJr>)rr*) r rrrinput_resolutionrrKrLrM)r$r^dimout_dimr(r*r+s rrPatchMerging.__init__s` Vv%'EF  *+; <C! C Ga7b Ga0 rcURS:Xa,URSS5RSUR5nUR UR UR U555nURS5RSS5nU$)Nr>rr=)ndim transpose unflattenr^rMrLrKflattenrSs rrTPatchMerging.forwardsn 66Q; Aq!++At/D/DEA JJtzz$**Q-0 1 IIaL " "1a (r)rKrLrMr^) r^int | tuple[int, int]r_rr`rr(rr*r-r.r/rWrYr8s@rr[r[vs_')gg 1/ 1 1 1  1 $ 1  1 1rr[cz^\rSrSrSr\R SSSS4S U4SjjjrS SjrS r U=r $) ConvLayera1Implement a convolutional layer with optional checkpointing and downsample. Args: dim: The number of input channels. depth: The number of blocks in the convolutional layer. activation: The activation function to use. Default: :class:`nn.GELU`. drop_path: The dropout rate for the stochastic depth. Default: 0.0. downsample: The downsample module to use. Default: None. use_checkpoint: Whether to use checkpointing for memory efficiency. Default: False. conv_expand_ratio: The expansion ratio for the hidden dimension. Default: 4.0. rHNF@c >[T U]5 X`l[U[5(dU/U-n[ R "[U5Vs/sHn[XXsXH5PM sn5Ul XPl gs snfr) rruse_checkpointrlistr ModuleListrangerFblocks downsample) r$r_depthr*rNrsrnconv_expand_ratioir+s rrConvLayer.__init__st ,)T**" e+ImmTYZ_T` aT`qVC/Y\ JT` a  % bsA:cURH2nUR(a[R"X!5OU"U5nM4 URbURU5nU$rrrrnrrsr$rblks rrTConvLayer.forwardK;;C151D1D %%c-#a&A ?? &"Arrrrsrn)r_rrtrr*r-rNfloat | list[float]rsOptional[nn.Module]rnboolrurVr.r/rWrYr8s@rrjrjs  ')gg),*.$#&% %%$ % ' % ( %%!% %%.rrjcb^\rSrSrSr\R S4SU4SjjjrSrU=r $)MLPaPImplement a multi-layer perceptron (MLP) with optional dropout. Args: in_features: The number of input features. hidden_features: The number of hidden features. out_features: The number of output features. activation: The activation function to use. Default: :class:`nn.GELU`. drop: The dropout rate. Default: 0.0. rHcH>[TU]5 [R"U5Ul[R "X5UlU"5Ul[R"U5Ul [R "X#5Ul [R"U5Ul gr) rrr LayerNormnormLinearfc1act1Dropoutdrop1fc2drop2)r$ in_featureshidden_features out_featuresr*dropr+s rr MLP.__init__si LL- 99[:L ZZ% 99_;ZZ% r)rrrrrr) rrrrrrr*r-rrVr.r/rCr8s@rrrsX')gg &&& & $ &  & &&rrc^\rSrSrSrSS U4Sjjjr\S Sj5r\R"5S S U4Sjjj5r S Sjr Sr U=r $) AttentionajImplement an attention mechanism with optional relative position encoding. Args: dim: The number of input channels. key_dim: The dimension of the key. num_heads: The number of attention heads. Default: 8. attn_ratio: The ratio of the attention dimension. Default: 4.0. resolution: The resolution of the input feature map. Default: (14, 14). cb>[T U]5 X0lUS-UlX lX#-Ul[ XB-5Ul[ XB-5U-UlX@l URUR S--n[R"U5Ul [R"X5Ul[R"URU5UlUR!U5upx[R""[$R&"X855UlUR+SUSS9 U SUlg)Ngr=attention_bias_idxsF) persistent)rr num_headsscalekey_dimnh_kdrddh attn_ratiorrrrqkvprojbuild_attention_bias Parametertorchzerosattention_biasesregister_bufferab) r$r_rrr resolutionhindicesattn_offset_sizer+s rrAttention.__init__s "d]  ( Z)*j*+i7$ GGdjj1n $LL% 99S$IIdggs+ $($=$=j$I! " U[[-U V 2GN *.rcUup[R"U5n[R"U5nURU5nURU5nUSS2S4USSS24- R 5nUSS2S4USSS24- R 5nXr-U-n [R "U SS9upU R X-X-5n U R5n X4$)NT)return_inverse)rarangerepeat_interleaverepeatabsuniqueviewnumel)rHWrowscolsrrccdrdckeys unique_keysinverserrs rrAttention.build_attention_bias s||A||A  # #A & [[^DkBtQwK' , , .DkBtQwK' , , .v{$||DF ,,quae,&,,.((rc>[TU]U5 U(aURb SUlU$URSS2UR4UlU$r)rtrainrrr)r$moder+s rrAttention.trainsM  dDGG$7$ ?C>S>STUW[WoWoTo>p rcURup#nURU5nURU5nURX#URS5R SSSS5nUR URURUR/SS9upgnUR(aURSS2UR4O URn XgRSS5-UR-U -n U RSS9n X-RSS5R!X#UR"5nUR%U5nU$)Nrr=rr>)r_)shaperrrrpermutesplitrrtrainingrrrrdrsoftmaxreshaperr) r$rBN_rqkvrattns rrTAttention.forwards ''a IIaLhhqkhhqT^^R088Aq!D))T\\4<<@a)HaEI]]t$$Q(@(@%@AX\X_X_KKB''4::5<|||# X A & . .qTWW = IIaLr) rrrrrrrrrrrr)rl)r) r_rrrrrrrVrtuple[int, int]r.r/)rrr.ztuple[torch.Tensor, int])T)rrr.rrW)r0r1r2r3r4r staticmethodrrno_gradrrTr6r7r8s@rrrs&. / // /  / $ / //8 ) ) ]]_  rrc^\rSrSrSrSSSSS\R 4S U4SjjjrS SjrS r U=r $) TinyViTBlocki/aoImplement a single block of the TinyViT architecture. This block consists of multi-head self-attention and a feed-forward network with residual connections. Args: dim: The input dimension size. input_resolution: The height and width of the input feature map. num_heads: The number of attention heads. window_size: The size of the local attention window. mlp_ratio: The ratio of MLP hidden dimension to embedding dimension. drop: The dropout rate. Default: 0.0. drop_path: The stochastic depth rate. Default: 0.0. layer_scale_init_value: The initial value for layer scaling. Default: 1e-5. rlrHr>c J>[X-S:HS5 [T U] 5 [U5UlX@lX-n [ XUSXD45Ul[U5Ul [XUSUS-U5Ul [U[X-5XU5Ul[U5Ulg)Nrz!dim must be divislbe by num_headsg?rr=)r rrrr^ window_sizerrr drop_path1r local_convrrmlp drop_path2) r$r_r^rr mlp_ratiorrNlocal_conv_sizer*head_dimr+s rrTinyViTBlock.__init__@s S_)+NO  *+; <&#cYk=WX "9- ?ARS?SUXYsC0#4H"9-rc@URup#URupEnUnURXBX65n[XR5upUR UR SS55n[XRXU45nURXEU5nXpRU5-nURSS5RXFX#5nURU5nURXFU5RSS5nXRURU55-nU$)Nrr=)r^rrr rrrfr rrdrrrr) r$rrrrLCres_xpad_hws rrTTinyViTBlock.forwardXs$$''a FF1 $Q(8(89  IIaii1o & q"2"2FF C FF1O OOA& & KK1  % %aA 1 OOA  FF1O % %a +  , ,r)rrrr^rrr)r_rr^rhrrrrrrVrrVrNrVrrr*r-r.r/rWrYr8s@rrr/s* &(gg. .0. .  .  ....$. ..0rrc^\rSrSrSrSSSSSS\R 4S U4SjjjrS S jrS r U=r $) BasicLayerimaoImplement a basic layer of the TinyViT architecture. This layer consists of a series of TinyViT blocks, each followed by a patch merging operation. Args: dim: The input dimension size. input_resolution: The height and width of the input feature map. depth: The number of blocks in the layer. num_heads: The number of attention heads. window_size: The size of the local attention window. mlp_ratio: The ratio of MLP hidden dimension to embedding dimension. drop: The dropout rate. Default: 0.0. drop_path: The stochastic depth rate. Default: 0.0. downsample: The downsample module to use. Default: None. use_checkpoint: Whether to use checkpointing for memory efficiency. Default: False. local_conv_size: The size of the local convolution kernel. Default: 3. activation: The activation function to use. Default: :class:`nn.GELU`. rlrHNFr>c >[TU]5 Xl[R"[ U5V s/sH/n [ UUUUUU[U[5(aXOUU U 5 PM1 sn 5Ul Xl gs sn fr) rrrnrrprqrrrorrrs)r$r_r^rtrrrrrNrsrnrr*rvr+s rrBasicLayer.__init__s ,mmu &A$$.y$$?$?ILY# &   $%# s6A<cURH2nUR(a[R"X!5OU"U5nM4 URbURU5nU$rryrzs rrTBasicLayer.forwardr}rr~)r_rr^rhrtrrrrrrrVrrVrNrrsrrnrrrr*r-r.r/rWrYr8s@rrrms6),*.$ &(gg$% $%0$% $%  $%  $%$%$%'$%($%$%$%$$% $%$%Lrrc^\rSrSrSrSSSSSSS S S S S S S\R S 4SU4S jjjrSSjr\ SSSjj5r Sr U=r $)TinyViTiaTinyViT model, as described in https://arxiv.org/abs/2207.10666. Args: img_size: Size of input image. in_chans: Number of input image's channels. num_classes: Number of output classes. embed_dims: List of embedding dimensions. depths: List of block count for each downsampling stage num_heads: List of attention heads used in self-attention for each downsampling stage. window_sizes: List of self-attention's window size for each downsampling stage. mlp_ratio: Ratio of MLP dimension to embedding dimension in self-attention. drop_rate: Dropout rate. drop_path_rate: Stochastic depth rate. use_checkpoint: Whether to use activation checkpointing to trade compute for memory. mbconv_expand_ratio: Expansion ratio used in MBConv block. local_conv_size: Kernel size of convolution used in TinyViTBlock activation: activation function. mobile_same: Whether to use modifications for MobileSAM. r>i)`ir=r=r=)r>r rrrrrlrHFc 8>[T U]5 XlXlU U(ad/SQn[R "[R "USSSSS9[S5[R "SSSSSSS9[S55UlO /SQnSUl[X$S U5Ul US -n[R"S U [U55Vs/sHnUR5PM nn[U5n/n[![#XEXgU55HununnnnnU[%US-[U5S- 5nUUS- :a['UUUUU5OSnUUU[USU5[USUS-5UU US .nUS :Xa[)SS U 0UD6nO[+SUUUUU U S .UD6nUR-U5 UU-nM [R "U6UlUUl[R2"US5Ul[R6"USU5Ulgs snf)N)r=r=rrrrF)rr>)r=r=r=rr)r_rtrNrsrnr*ru)r^rrrrr)rrimg_size mobile_samrr?rr neckr: patch_embedrlinspacesumitemlen enumeratezipminr[rjrappendlayers feat_sizer norm_headrhead)!r$rin_chans num_classes embed_dimsdepthsr window_sizesr drop_ratedrop_path_raternmbconv_expand_ratiorr*rstridesr^rdprn_layersri_layerrArt num_heads_irr(r`rskwargslayerr+s! rrTinyViT.__init__s?&  $ #G  *R.#qu=C  #sAq!%8C DI#GDI%h1 zJ#q="'>3v;!OP!OAqvvx!OPv;MV  IW EN IGIi [&!Wq[#j/A2E!FGGhl*-y'6:V  ! VHW%5!6VMgPQk=R9ST("0( &F!|!R4GR6R"%5) +'"$3  MM%  ' ?N @mmV, ) jn5IIjnk: [QsHchURU5nURU5nUR(aMURSURUR45R SSSS5nUR U5nU$URS5nURURU55nU$)zUClassify images if ``mobile_sam=False``, produce feature maps if ``mobile_sam=True``.rrr>r=) r rrrerrrmeanrrrSs rrTTinyViT.forwards   Q  KKN ?? A?@HHAqRSTA ! A q A $..+,Arc X[US;S5 [[[S.U"U40UD6$)aCreate a TinyViT model from pre-defined variants. Args: variant: TinyViT variant. Possible values: ``'5m'``, ``'11m'``, ``'21m'``. pretrained: whether to use pre-trained weights. Possible values: ``False``, ``True``, ``'in22k'``, ``'in1k'``. For TinyViT-21M (``variant='21m'``), ``'in1k_384'``, ``'in1k_512'`` are also available. **kwargs: other keyword arguments that will be passed to :class:`TinyViT`. .. note:: When ``img_size`` is different from the pre-trained size, bicubic interpolation will be performed on attention biases. When using ``pretrained=True``, ImageNet-1k checkpoint (``'in1k'``) is used. For feature extraction or fine-tuning, ImageNet-22k checkpoint (``'in22k'``) is preferred. )5m11m21mz+Only variant 5m, 11m, and 21m are supported)r _tiny_vit_5m _tiny_vit_11m _tiny_vit_21m)variant pretrainedr#s r from_configTinyViT.from_config.s4 W 446cd"=OPWXYcngmnnr)rrrrrrrr ) rrrrrrr Sequence[int]rr4rr4rr4rrVrrVrrVrnrrrVrrr*r-rrr.r/rWF)r0strr1 bool | strr#rr.r) r0r1r2r3r4rrDrrTrr2r6r7r8s@rrrs.$7 ,#1&3 #$%( &(gg #W;W;W; W; " W;  W;!W;$W;W;W;W;W;#W;W; $!W;"#W;$ %W;W;r oorrcjUR5n[RRU5nSU;aUSnUR 5Vs/sH nSU;dM UPM nnUHnX4R upgX$R up[ Xh:HSUSU35 Xy:wdM>[US-5n [U S-5n X4RSXjU 5n [R"XU 4SS9n U RX5X4'M US R S URR:waqS n [R"U SS 9 [R"URR 5US '[R"URR"5US 'UR%U5 U$s snf)Nmodelrz Fail to load z/. Pre-trained checkpoint should have num_heads=g?rbicubic)sizerz head.weightrzaNumber of classes does not match pre-trained checkpoint's. Resetting classification head to zeros) stacklevelz head.bias) state_dictrhubload_state_dict_from_urlrrr rrF interpolaterrwarningswarn zeros_likeweightrload_state_dict)r9urlmodel_state_dictr=rab_keysn_heads1L1n_heads2L2S1S2rmsgs r_load_pretrainedrQBs'')33C8J*( %//+G+Q/AQ/Fq+GG !}** '*00 X)]1#=lmulv+wx 8RWBRWB)}11!X2F  }}-=HS\] ,11(?JM- &&q)UZZ-D-DDq ca($)$4$4UZZ5F5F$G =!"'"2"25::??"C ; *% L)Hs F0F0c z[S /SQ/SQ/SQ/SQSS.UD6nU(aUSLaSnS S S .Un[X#5nU$) N)@i@r)r=r rrHrrrrrTin1kzchttps://github.com/wkcn/TinyViT-model-zoo/releases/download/checkpoints/tiny_vit_5m_22k_distill.pthzghttps://github.com/wkcn/TinyViT-model-zoo/releases/download/checkpoints/tiny_vit_5m_22kto1k_distill.pthin22krYrrrQr1r#r9rGs rr-r-csg  &"     E  Jv}    !, Lrc z[S /SQ/SQ/SQ/SQSS.UD6nU(aUSLaSnS S S .Un[X#5nU$) N)rSrTrir)r=rrrrg?rXTrYzdhttps://github.com/wkcn/TinyViT-model-zoo/releases/download/checkpoints/tiny_vit_11m_22k_distill.pthzhhttps://github.com/wkcn/TinyViT-model-zoo/releases/download/checkpoints/tiny_vit_11m_22kto1k_distill.pthrZrr\r]s rr.r.|sg  &"     E  Jw~    !, Lrc [S/SQ/SQ/SQ/SQSS.UD6nU(a>USLa$SnURS S 5nUS :aS nUS :aSnSSSSS.Un[X$5nU$)N)rrri@r)r>rrrg?rXTrYrrrin1k_384iin1k_512zdhttps://github.com/wkcn/TinyViT-model-zoo/releases/download/checkpoints/tiny_vit_21m_22k_distill.pthzhhttps://github.com/wkcn/TinyViT-model-zoo/releases/download/checkpoints/tiny_vit_21m_22kto1k_distill.pthzlhttps://github.com/wkcn/TinyViT-model-zoo/releases/download/checkpoints/tiny_vit_21m_22kto1k_384_distill.pthzlhttps://github.com/wkcn/TinyViT-model-zoo/releases/download/checkpoints/tiny_vit_21m_22kto1k_512_distill.pth)r[rYrarbr)rgetrQ)r1r#r9rrGs rr/r/s  & "     E  Jzz*c2H3' 3' w~GG   !, Lr)rrhr.r)r9rrGr6r.rr5)r1r7r#rr.r)& __future__rrBtypingrrrrtorch.nn.functionalr functionalr@ torch.utilsrkornia.core.checkr kornia.models.commonr r r r rr?rr:ModulerFr[rjrrrrrrQr-r.r/rrrrls*#** "*\\/ R]] 8  &SRYYS@299B) )X&"--&>J JZ;299;|@@FOobiiOodB22r