3jW, SSKJrJrJr SSKrSSKJr S\S\S\R4SjrSS\S\S\S \S\R4 S jjr "S S \R5r "S S\R5r "SS\R5r "SS\R5r "SS\R5r"SS\R5r"SS\R5rg))AnyDictTupleN)nninpoupreturnc [R"[R"XSSSSS9[R"U5[R"55$)z&Apply 1x1 Convolution with Batch Norm.rFbiasr SequentialConv2d BatchNorm2dSiLU)rrs R/home/wildlama/miniconda3/lib/python3.13/site-packages/kornia/models/vit_mobile.py conv_1x1_bnrs: ==3Q15A2>>RUCVXZX_X_Xa bb kernal_sizestridec [R"[R"XX#SSS9[R"U5[R"55$)z&Apply NxN Convolution with Batch Norm.r Fr r)rrrrs r conv_nxn_bnrs; ==3[!%PRTR`R`adRegigngngp qqrc^\rSrSrSrS\S\RSS4U4SjjrS\ RS \ \ \ 4S\ R4S jrS rU=r$) PreNorm"zApply a normalization layer before the functional block. Args: dim: The input dimension size. fn: The module or function to be executed after normalization. dimfnr Ncd>[TU]5 [R"U5UlX lgN)super__init__r LayerNormnormr)selfrr __class__s rr"PreNorm.__init__*s# LL% rxkwargsc FUR"URU540UD6$r rr$)r%r(r)s rforwardPreNorm.forward/swwtyy|.v..rr+)__name__ __module__ __qualname____firstlineno____doc__intrModuler"torchTensorrstrrr,__static_attributes__ __classcell__r&s@rrr"sUCRYY4 //c3h/ELL//rrc ~^\rSrSrSrS S\S\S\SS4U4SjjjrS \RS\R4S jr S r U=r $) FeedForward3zImplement the Feed-Forward network block for Vision Transformers. Args: dim: The input dimension size. hidden_dim: The dimension of the hidden layer. dropout: The dropout probability. Default: 0.0. r hidden_dimdropoutr Nc &>[TU]5 [R"[R"X5[R "5[R "U5[R"X!5[R "U55Ulgr )r!r"rrLinearrDropoutnet)r%rr>r?r&s rr"FeedForward.__init__<sX == IIc & 2::g3F R\Hbdfdndnovdw rr(c$URU5$r rCr%r(s rr,FeedForward.forwardBsxx{rrF r.r/r0r1r2r3floatr"r5r6r,r8r9r:s@rr<r<3sN C S 5 4  %,,rr<c ^\rSrSrSrS S\S\S\S\SS4 U4S jjjrS \RS\R4S jr S r U=r $) AttentionFzImplement the Multi-Head Attention module. Args: dim: The input dimension size. heads: The number of attention heads. Default: 8. dim_head: The dimension of each head. Default: 64. dropout: The dropout probability. Default: 0.0. rheadsdim_headr?r Nc>[TU]5 X2-nUS:H=(a X1:H(+nX lUS-Ul[R "SS9Ul[R"XS-SS9UlU(aE[R"[R"XQ5[R"U55Ul g[R"5Ul g)Nr grFr ) r!r"rPscalerSoftmaxattendrAto_qkvrrBIdentityto_out)r%rrPrQr? inner_dim project_outr&s rr"Attention.__init__Ps $  A:9(/:  t^ jjR( ii]? WbbmmBIIi$=rzz'?RS hjhshshu rr(c^^ ^ ^ ^ TRU5RSSS9nUSRum m m m U U U U U4SjU5up4n[R"X4R SS55TR -nTRU5n[R"Xu5nUR SS5RT T T T 5nTRU5$)NrUrSrTrc 3># UH@oRTTTTRTTR-5RSS5v MB g7f)rUN)reshaperP transpose).0tbhdnpr%s r $Attention.forward..asAa]`XY99Q1djj" 2BCMMaQRSS]`sAA ra) rYchunkshaper5matmulrcrVrXrbr[) r%r(qkvqkvdotsattnoutrfrgrhris ` @@@@rr,Attention.forward]skk!n""1""-!fll 1aaa]`aa||A{{2r23djj@{{4 ll4#mmAq!))!Q26{{3r)rXrPrVr[rY)@rJrKr:s@rrNrNFs] vC v v3 ve v^b v v    %,,   rrNc^\rSrSrSrSS\S\S\S\S\S\S S 4U4S jjjrS \RS \R4S jr Sr U=r $) TransformerjazTransformer block described in ViT. Paper: https://arxiv.org/abs/2010.11929 Based on: https://github.com/lucidrains/vit-pytorch Args: dim: input dimension. depth: depth for transformer block. heads: number of heads in multi-head attention layer. dim_head: head size. mlp_dim: dimension of the FeedForward layer. dropout: dropout ratio, defaults to 0. rdepthrPrQmlp_dimr?r Nc.>[TU]5 [R"/5Ul[ U5H\nURR [R"[U[XXF55[U[XU55/55 M^ gr ) r!r"r ModuleListlayersrangeappendrrNr<) r%rr}rPrQr~r?_r&s rr"Transformer.__init__zsq mmB' uA KK   Ys8%MN[w%GH rr(c\URHup#U"U5U-nU"U5U-nM U$r r)r%r(ruffs rr,Transformer.forwards3 HDQ! A1 A$rrrIrKr:s@rr{r{jsf  C  C 3 QT _d os  %,,rr{c ^\rSrSrSrS S\S\S\S\SS4 U4S jjjrS \RS\R4S jr S r U=r $)MV2BlockaQMV2 block described in MobileNetV2. Paper: https://arxiv.org/pdf/1801.04381 Based on: https://github.com/tonylins/pytorch-mobilenet-v2 Args: inp: input channel. oup: output channel. stride: stride for convolution, defaults to 1, set to 2 if down-sample. expansion: expansion ratio for hidden dimension, defaults to 4. rrr expansionr Nc0>[TU]5 X0l[X-5nURS:H=(a X:HUlUS:Xa[ R "[ R"XUSUSUSS9[ R"U5[ R"5[ R"XRSSSSS9[ R"U55Ul g[ R "[ R"XSSSSS9[ R"U5[ R"5[ R"XUSUSUSS9[ R"U5[ R"5[ R"XRSSSSS9[ R"U55Ul g)Nr rUF)groupsr rr ) r!r"rr3use_res_connectrrrrrconv)r%rrrrr>r&s rr"MV2Block.__init__s!  ) #{{a/>CJ >  *!VQzX]^z*  *1a?s#DI  #1a?z*  *!VQzX]^z*  *1a?s# DIrr(clUR(aXRU5-$URU5$r )rrrGs rr,MV2Block.forwards*   yy|# #99Q< r)rrr)r ) r.r/r0r1r2r3r"r5r6r,r8r9r:s@rrrsW Cc3sSW@  %,,  rrc^\rSrSrSrSS\S\S\S\S\\\4S\S \S S 4U4S jjjrS \ RS \ R4Sjr Sr U=r $)MobileViTBlockaUMobileViT block mentioned in MobileViT. Args: dim: input dimension of Transformer. depth: depth of Transformer. channel: input channel. kernel_size: kernel size. patch_size: patch size for folding and unfloding. mlp_dim: dimension of the FeedForward layer in Transformer. dropout: dropout ratio, defaults to 0. rr}channel kernel_size patch_sizer~r?r Nc>[TU]5 UuUlUl[ X3U5Ul[ X15Ul[XSSXg5Ul [ X5Ul [ SU-X45Ul g)Nrrxra) r!r"phpwrconv1rconv2r{ transformerconv3conv4) r%rr}rrrr~r?r&s rr"MobileViTBlock.__init__si % ;?  . &s1aJ .  WgC rr(cUR5nURU5nURU5nURup4pVXPR-X`R -pUR X4-U-URXR 5RSS5nUR X4Xx-URUR -5RSS5nURU5nURSS5R X4-U-XRUR 5nURSS5R X4XV5nURU5n[R"X4S5nURU5nU$)Nr rarU) clonerrrnrrrbrcrrr5catr) r%r(yrfdhwnhnws rr,MobileViTBlock.forwards5 GGI JJqM JJqMWW aggqGG|B IIaebj$''2ww 7 A A!Q G IIaBGTWWtww%6 7 A A!Q G   Q  KK1  % %aebj"ggtww G KK1  % %aA 1 JJqM IIqfa  JJqMr)rrrrrrrrI)r.r/r0r1r2r3rrLr"r5r6r,r8r9r:s@rrrs *D DD D  D #s(O DDD DD*%,,rrc ^\rSrSrSrS S\S\S\\\4S\SS4 U4S jjjr S \ RS\ R4S jr S r U=r$) MobileViTia Module MobileViT. Default arguments is for MobileViT XXS. Paper: https://arxiv.org/abs/2110.02178 Based on: https://github.com/chinhsuanwu/mobilevit-pytorch Args: mode: 'xxs', 'xs' or 's', defaults to 'xxs'. in_channels: the number of channels for the input image. patch_size: image_size must be divisible by patch_size. dropout: dropout ratio in Transformer. Example: >>> img = torch.rand(1, 3, 256, 256) >>> mvit = MobileViT(mode='xxs') >>> mvit(img).shape torch.Size([1, 320, 8, 8]) mode in_channelsrr?r Nc B>[T U]5 US:Xa Sn/SQn/SQnO!US:Xa Sn/SQn/SQnOUS :Xa Sn/S Qn/S QnS n/S Qn [UWSSS9Ul[R "/5UlUR R[USUSSW55 UR R[USUSSU55 UR R[USUS SU55 UR R[USUS SU55 UR R[US USSU55 UR R[USUSSU55 UR R[USUSSU55 [R "/5Ul URR[WSU SUSX[USS-5US95 URR[USU SUSX[USS-5US95 URR[USU SUSX[USS-5US95 [USUS5Ul g)Nxxsra)ryP`) rr0rryryrri@xsr)rx) r rrryryrrrris)r) rrryryrrrrirU)rarrUr)rr rx)r? rlrS)r!r"rrrrmv2rrmvitrr3rr) r%rrrr?rdimschannelsrr}r&s rr"MobileViT.__init__sm  5=IDDH T\I!DDH S[I"DHH  hqk!D ==$ !hqk1iHI !hqk1iHI !hqk1iHI !hqk1iHI !hqk1iHI !hqk1iHI !hqk1iHIMM"%   47E!Hhqk;TWX\]^X_bcXcTdnu v   47E!Hhqk;TWX\]^X_bcXcTdnu v   47E!Hhqk;TWX\]^X_bcXcTdnu v !"x|< rr(cURU5nURS"U5nURS"U5nURS"U5nURS"U5nURS"U5nURS"U5nURS"U5nURS"U5nURS"U5nURS"U5nURU5nU$)Nrr rarUrrr)rrrrrGs rr,MobileViT.forwardDs JJqM HHQKN HHQKN HHQKN HHQKN HHQKN IIaLO HHQKN IIaLO HHQKN IIaLO JJqMr)rrrr)rrU)rararJ)r.r/r0r1r2r7r3rrLr"r5r6r,r8r9r:s@rrrsj(or*=*=.1*=CHc?*=fk*= *=*=X%,,rr)rUr )typingrrrr5rr3r4rrrr<rNr{rrrrrrs$$# cScscryyc rSrsrr#rbiir /bii/""))&!  ! H!"))!H2 ryy2 j>RYY>BQ Qr