3j-SrSSKJr SSKJrJr SSKrSSKJr SSKJ r S/r "SS \R5r "S S \R5r"S S \R5r"SS\R5r"SS\R5r"SS\R5r"SS\R5r/SQrSSjrg)aeModule that implement Vision Transformer (ViT). Paper: https://paperswithcode.com/paper/an-image-is-worth-16x16-words-transformers-1 Based on: `https://towardsdatascience.com/implementing-visualttransformer-in-pytorch-184f9f16f632` Added some tricks from: `https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/vision_transformer.py` ) annotations)AnyCallableN)nn) KORNIA_CHECKVisionTransformerc6^\rSrSrSU4SjjrSSjrSrU=r$) ResidualAdd'c.>[TU]5 XlgN)super__init__fn)selfr __class__s K/home/wildlama/miniconda3/lib/python3.13/site-packages/kornia/models/vit.pyrResidualAdd.__init__(s c 8UnUR"U40UD6nX- nU$r r)rxkwargsress rforwardResidualAdd.forward,s& GGA   rr)rzCallable[..., torch.Tensor]returnNone)r torch.Tensorrrrr__name__ __module__ __qualname____firstlineno__rr__static_attributes__ __classcell__rs@rr r 'srr c0^\rSrSrSSU4SjjjrSrU=r$) FeedForward3c >[TU][R"X5[R"5[R "U5[R"X#5[R "U55 gr )rrrLinearGELUDropout)r in_featureshidden_features out_features dropout_raters rrFeedForward.__init__4sI  IIk 3 GGI JJ| $ IIo 4 JJ| $  r)) r/intr0r6r1r6r2floatrrr!r"r#r$rr%r&r's@rr)r)3s   rr)c6^\rSrSrSU4SjjrSSjrSrU=r$)MultiHeadAttention>c>[TU]5 XlX lX-nUS-UlURUR-(a&[ SURSURS35e[ R"XS-5Ul[ R"U5Ul [ R"X5Ul [ R"U5Ul g)NgzySize of embedding inside the transformer decoder must be visible by number of headsfor correct multi-head attention Got: z embedding size and z numbers of heads) rremb_size num_headsscale ValueErrorrr,qkvr.att_drop projectionprojection_drop)rr>r?rC proj_drop head_sizers rrMultiHeadAttention.__init__?s  ") _ ==4>> ) &:4>>:JJ[] 99X!|4 8, ))H7!zz)4rc6URup#nURU5RX#SURX@R-5R SSSSS5nUSUSUSpn[ R "SXg5UR-n U RSS9n URU 5n [ R "S X5n U R SSSS5R5RX#S5n URU 5n URU 5n U $) Nr=rzbhqd, bhkd -> bhqkdimzbhal, bhlv -> bhav )shaperBreshaper?permutetorcheinsumr@softmaxrC contiguousviewrDrE) rrBNCrBqkvattouts rrMultiHeadAttention.forwardSs''ahhqk!!!4>>1;NOWWXY[\^_abdefa&#a&#a&all/6Ckkbk!mmC ll0#9kk!Q1%00277bAooc"""3' r)rCr>r?rDrErBr@) r>r6r?r6rCr7rFr7rrrrrrr r's@rr:r:>s5(rr:c,^\rSrSrSU4SjjrSrU=r$)TransformerEncoderBlockhcp>[TU][[R"[R "US5[ XXC5[R"U555[[R"[R "US5[XS-XS9[R"U5555 g)Nư>rL)r2) rrr r Sequential LayerNormr:r.r))r embed_dimr?r2 dropout_attnrs rr TransformerEncoderBlock.__init__is   LLD1&y\XJJ|,   LLD1 q=)_JJ|,  rr4) rir6r?r6r2r7rjr7rrr8r's@rrcrchs   rrccZ^\rSrSrSSU4SjjjrSSjrSrU=r$)TransformerEncoder|c>^^^^[TU]5 [R"UUUU4Sj[ U556Ul/Ulg)Nc3@># UHn[TTTT5v M g7fr )rc).0_rjr2rir?s r .TransformerEncoder.__init__..s# nam\]%iL,WWams)rrrrgrangeblocksresults)rridepthr?r2rjrs ` ```rrTransformerEncoder.__init__}s8 mm nafglam n  ,. rc/UlUnURR5H&nU"U5nURRU5 M( U$r )rwrvchildrenappend)rrr_ms rrTransformerEncoder.forwardsG %%'AC&C LL   $( r)rvrw) rr5r5) rir6rxr6r?r6r2r7rjr7rrrar r's@rrmrm|sb!! . . . .  .  .  . .rrmch^\rSrSrSrSSU4SjjjrS SjrS SjrSrU=r $) PatchEmbeddingzJCompute the 2d image patch embedding ready to pass to transformer encoder.c>[TU]5 XlX lX0lU=(d [ R "XX3S9UlUbURXU45up&X lOXC-S-n[ R"[R"SSU55Ul [ R"[R"US-U55Ul g)N) kernel_sizestriderJrK)rr in_channels out_channels patch_sizerConv2dbackbone_compute_feats_dims ParameterrSrandn cls_token positions)rrrr image_sizer feat_sizers rrPatchEmbedding.__init__s &($!sBIIkU_$s  &*&>&> Yc?d&e #L , #1a7Iekk!Q &EFekk)a-&NOrcUR[R"S/UQ765R5nURSURSURS-4$)NrKrM)rrSzerosdetachrP)rrr_s rr"PatchEmbedding._compute_feats_dimssOmmEKK7J78??Ayy}ciimciim;;;rcURU5nURup# nURX#S5RSSS5nURR USS5n[ R"XQ/SS9nXR- nU$)NrMrrJrKrN) rrPrWrRrrepeatrScatr)rrrXrYrr cls_tokenss rrPatchEmbedding.forwardsy MM! WW a FF1  $ $Q1 -^^**1a3 IIzo1 - ^^r)rrrrrr)r=rN) rr6rr6rr6rr6rnn.Module | Nonerr)rztuple[int, int, int]rztuple[int, int]ra) r!r"r#r$__doc__rrrr%r&r's@rrrsrT%) PPP P  P # P PP0<  rrc^\rSrSrSrSS U4Sjjjr\S Sj5rS Sjr\ S S Sjj5r Sr U=r $)raVision transformer (ViT) module. The module is expected to be used as operator for different vision tasks. The method is inspired from existing implementations of the paper :cite:`dosovitskiy2020vit`. .. warning:: This is an experimental API subject to changes in favor of flexibility. Args: image_size: the size of the input image. patch_size: the size of the patch to compute the embedding. in_channels: the number of channels for the input. embed_dim: the embedding dimension inside the transformer encoder. depth: the depth of the transformer. num_heads: the number of attention heads. dropout_rate: dropout rate. dropout_attn: attention dropout rate. backbone: an nn.Module to compute the image patches embeddings. Example: >>> img = torch.rand(1, 3, 224, 224) >>> vit = VisionTransformer(image_size=224, patch_size=16) >>> vit(img).shape torch.Size([1, 197, 768]) c >[T U]5 XlX lX0lX@l[ X4X!U 5UlURRn [XXgU5Ul [R"U S5Ul g)Nrf)rrrrr embed_sizerpatch_embeddingrrmencoderrrhnorm) rrrrrirxr?r2rjr hidden_dimrs rrVisionTransformer.__init__sk $$&#-kj^fg))66 )*YVbc LLT2 rc.URR$r )rrw)rs rencoder_results!VisionTransformer.encoder_resultss||###rc [U[R5(d[S[ U535eUR /UR SSQ7;a\UR SUR:wa?[SURSUR SUR SUR 35eURU5nURU5nURU5nU$)Nz)Input x type is not a torch.Tensor. Got: rrzInput image shape must be Bxrz. Got: ) isinstancerSTensor TypeErrortyperrPrrArrr)rrr_s rrVisionTransformer.forwards!U\\**GQyQR R ??"2AGGBCL"2 2qwwr{dFVFV7V.t/?/?.@$//ARRSTXTcTcSddklmlslsktu ""1%ll3iin rc 8URS5up4[U5nSSSS.SSSS.SSSS.S S S S.S S S S.S.UnURXeS9 [S0UD6nU(a;[ U5n[ R RU5n URU 5 U$)a0Build ViT model based on the given config string. The format is ``vit_{size}/{patch_size}``. E.g. ``vit_b/16`` means ViT-Base, patch size 16x16. If ``pretrained=True``, AugReg weights are loaded. The weights are hosted on HuggingFace's model hub: https://huggingface.co/kornia. .. note:: The available weights are: ``vit_l/16``, ``vit_b/16``, ``vit_s/16``, ``vit_ti/16``, ``vit_b/32``, ``vit_s/32``. Args: variant: ViT model variant e.g. ``vit_b/16``. pretrained: whether to load pre-trained AugReg weights. kwargs: other keyword arguments that will be passed to :func:`kornia.models.vit.VisionTransformer`. Returns: The respective ViT model Example: >>> from kornia.models.vit import VisionTransformer >>> vit_model = VisionTransformer.from_config("vit_b/16", pretrained=True) /rr=)rirxr?iriri )vit_tivit_svit_bvit_lvit_h)rr4) splitr6updater_get_weight_urlrShubload_state_dict_from_urlload_state_dict) variant pretrainedr model_typepatch_size_strr model_configmodelurl state_dicts r from_configVisionTransformer.from_configs2&-]]3%7" ( %("1E#&!D#&"E#'"2F#'"2F      l :!+F+ !'*C;;C@J  ! !* - r)rrrrrrr) rrr=rrrr5r5N)rr6rr6rr6rir6rxr6r?r6r2r7rjr7rrrr)rzlist[torch.Tensor]ra)F)rstrrboolrrrr) r!r"r#r$rrpropertyrr staticmethodrr%r&r's@rrrs<!!%)333 3  3  3333#3 33.$$ ++r)zvit_l/16zvit_b/16zvit_s/16z vit_ti/16zvit_b/32zvit_s/32cr[U[;SUS35 URS5upSUUSUSUS3$)z$Return the URL of the model weights.zVariant z% does not have pre-trained checkpointrzhttps://huggingface.co/kornia/z_augreg_i21k_r224/resolve/main/-z.pth)r_AVAILABLE_WEIGHTSr)rrrs rrr7sY..(7)Ch0ij$]]3/J +J< |Cbcmbnnopzo{{ AAr)rrrr)r __future__rtypingrrrSrkornia.core.checkr__all__Moduler rgr)r:rcrmrrrrr4rrrs$# *   "))  "-- ''T bmm (0(RYY(Vr rj_Ar