3j SrSSKrSSKrSSKJr SSKJrJrJrJ r J r J r J r J r SSKrSSKJr SSKJs Jr SSKJrJr SSKJrJrJrJrJrJrJr SSKJ r SS K!J"r" SS K#J$r$ SS K%J&r&J'r' SS K(J)r)J*r* S /r+\RX"\-5r.S\R^S\R^4Sjr0S\R^S\R^4Sjr1S\R^S\ \2\24S\R^4Sjr3\$S\R^S\ \2\24S\ \2\24S\R^4Sj5r4"SS\Rj5r6"SS\Rj5r7"SS\Rj5r8"SS\Rj5r9"SS \Rj5r:"S!S \Rj5r;SZS#\RjS$\S'r?S[S(jr@S\S)\S-S?S/9_S@\A"S"S,S-S.S/9_SA\A"S"S1S2S39_SB\A"S"S1S2S39_SC\A"S"S,S-S.S/9_SD\A"S"S1S2S39_SE\A"S"S,S-S.S/9_SF\A"S"S1S2S39_SG\A"S"S,S-S.S/9_SH\A"S"S1S2S39_5rB\*S[SI\=S\;4SJjj5rC\*S[SI\=S\;4SKjj5rD\*S[SI\=S\;4SLjj5rE\*S[SI\=S\;4SMjj5rF\*S[SI\=S\;4SNjj5rG\*S[SI\=S\;4SOjj5rH\*S[SI\=S\;4SPjj5rI\*S[SI\=S\;4SQjj5rJ\*S[SI\=S\;4SRjj5rK\*S[SI\=S\;4SSjj5rL\*S[SI\=S\;4STjj5rM\*S[SI\=S\;4SUjj5rN\*S[SI\=S\;4SVjj5rO\*S[SI\=S\;4SWjj5rP\*S[SI\=S\;4SXjj5rQ\*S[SI\=S\;4SYjj5rRg)]aSwin Transformer V2 A PyTorch impl of : `Swin Transformer V2: Scaling Up Capacity and Resolution` - https://arxiv.org/pdf/2111.09883 Code adapted from https://github.com/ChristophReich1996/Swin-Transformer-V2, original copyright/license info below This implementation is experimental and subject to change in manners that will break weight compat: * Size of the pos embed MLP are not spelled out in paper in terms of dim, fixed for all models? vary with num_heads? * currently dim is fixed, I feel it may make sense to scale with num_heads (dim per head) * The specifics of the memory saving 'sequential attention' are not detailed, Christoph Reich has an impl at GitHub link above. It needs further investigation as throughput vs mem tradeoff doesn't appear beneficial. * num_heads per stage is not detailed for Huge and Giant model variants * 'Giant' is 3B params in paper but ~2.6B here despite matching paper dim + block counts * experiments are ongoing wrt to 'main branch' norm layer use and weight init scheme Noteworthy additions over official Swin v1: * MLP relative position embedding is looking promising and adapts to different image/window sizes * This impl has been designed to allow easy change of image size with matching window size changes * Non-square image size and window size are supported Modifications and additions for timm hacked together by / Copyright 2022, Ross Wightman N)partial)AnyCallableDictListOptionalTupleTypeUnionIMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD)DropPathcalculate_drop_path_ratesMlpClassifierHead to_2tuple_assertndgrid)build_model_with_cfg)feature_take_indices)register_notrace_function) named_apply checkpoint)generate_default_cfgsregister_modelSwinTransformerV2Crxreturnc*URSSSS5$)z>Permutes a tensor from the shape (B, C, H, W) to (B, H, W, C).rrpermuters \/home/wildlama/miniconda3/lib/python3.13/site-packages/timm/models/swin_transformer_v2_cr.py bchw_to_bhwcr(4 99Q1a  c*URSSSS5$)z>Permutes a tensor from the shape (B, H, W, C) to (B, C, H, W).rr#rr"r$r&s r' bhwc_to_bchwr,9r)r* window_sizecURup#pEURX#US-USXAS-USU5nURSSSSSS5R5RSUSUSU5nU$)zPartition into non-overlapping windows. Args: x: Input tensor of shape (B, H, W, C). window_size: Window size (height, width). Returns: Windows tensor of shape (num_windows*B, window_size[0], window_size[1], C). rrr#r"shapeviewr% contiguous)rr-BHWCwindowss r'window_partitionr;>sJA! q{1~%{1~qN7JKXYN\]^Aii1aAq)446;;B AP[\]P^`abG Nr*r:img_sizecUup4URSnURSX1S-XAS-USUSU5nURSSSSSS5R5RSX4U5nU$)aMerge windows back to feature map. Args: windows: Windows tensor of shape (num_windows * B, window_size[0], window_size[1], C). window_size: Window size (height, width). img_size: Image size (height, width). Returns: Feature map tensor of shape (B, H, W, C). r1rrr#r"r/r0r2)r:r-r<r7r8r9rs r'window_reverser>Ns DA bA R!n,aq>.A;q>S^_`SacdeA !Q1a#..055b!BA Hr*c,^\rSrSrSrSS\S\S\\\4S\S\S \S \S S4U4S jjjr SS jr SSjr SSjr S\\\4S S4Sjr S \R4SjrSS\RS\\RS \R4SjjrSSjrSrU=r$)WindowMultiHeadAttentionaaThis class implements window-based Multi-Head-Attention with log-spaced continuous position bias. Args: dim (int): Number of input features window_size (int): Window size num_heads (int): Number of attention heads drop_attn (float): Dropout rate of attention map drop_proj (float): Dropout rate after projection meta_hidden_dim (int): Number of hidden features in the two layer MLP meta network sequential_attn (bool): If true sequential self-attention is performed Ndim num_headsr- drop_attn drop_projmeta_hidden_dimsequential_attnr c >XS.n [T U]5 X-S:XdS5eXl[U5UlX lXpl[R"S XS-SS.U D6Ul [R"U5Ul [R"S XSS.U D6Ul [R"U5Ul [SUU[RSS .U D6Ul[R""[$R&"U40U D65UlURupUR+S [$R&"X-U -U -S40U D6S S 9 UR-5 g)Ndevicedtyperz`The number of input features (in_features) are not divisible by the number of heads (num_heads).r#T in_features out_featuresbiasr")g?)hidden_featuresrN act_layerdroprelative_coordinates_logF persistent)r")super__init__rMrr-rCrGnnLinearqkvDropout attn_dropproj proj_droprReLUmeta_mlp Parametertorchempty logit_scaleregister_bufferreset_parameters)selfrBrCr-rDrErFrGrJrKddwin_hwin_w __class__s r'rY!WindowMultiHeadAttention.__init__nsO / !# o n o# #,5k,B'%499T7TQSTI.IIQ#dQbQ I.  +"gg     << I(D(DE''   & KK -5q ?B ?   r*c[RRUR[R "S55 UR 5 g)"Initialize parameters and buffers. N)rZinit constant_rfmathlog _init_buffersris r'rh)WindowMultiHeadAttention.reset_parameterss0 $**DHHRL9 r*c$UR5 g).Compute and fill non-persistent buffer values.N)"_make_pair_wise_relative_positionsrws r'rv&WindowMultiHeadAttention._init_bufferss //1r*c URRn[R"[ [R "UR SU[RS9[R "UR SU[RS955RS5nUSS2SS2S4USS2SSS24- nURSSS5RSS5R5n[R"U5[R"SUR5-5-nURR!UR#URR$55 g)zMInitialize the pair-wise relative positions to compute the positional biases.rrIrNr"r1?)rfrJrdstackraranger-float32flattenr%reshapefloatsignruabsrTcopy_torK)rirJ coordinatesrelative_coordinatesrTs r'r{;WindowMultiHeadAttention._make_pair_wise_relative_positionss"!!((kk& LL))!,V5== Q LL))!,V5== Q#   71:  +1a:6QaZ9PP3;;Aq!DLLRQRSYY[#(::.B#Ceii &**, ,G.$.  %%++,D,G,GHXHXH^H^,_`r*cf[U5nXR:waXlUR5 gg)zjUpdate window size and regenerate relative position coordinates. Args: window_size: New window size. N)rr-r{)rir-s r'set_window_size(WindowMultiHeadAttention.set_window_sizes1  , ** **   3 3 5 +r*cURSURS-nURUR5nURSS5R UR X5nUR S5nU$)zCompute the relative positional encodings. Returns: Relative positional encodings of shape (1, num_heads, window_size**2, window_size**2). rr)r-rbrT transposerrC unsqueeze)ri window_arearelative_position_biass r'_relative_positional_encodings7WindowMultiHeadAttention._relative_positional_encodingssy &&q)D,<,