3jKSSKJrJr SSKJr SSKrSSKJrJr SSKJ r SSK J r SSK JrJr SS KJr SS KJr SS KJrJrJrJrJr SS KJrJr SS KJr SSKJ r SSK!J"r"J#r#J$r$ SSK%J&r&J'r' SSK(J)r) SSK*J+r+ \#"SS9\"SS\555r,"SS\RZ5r."SS\RZ5r/"SS\RZ5r0SOS\RZS\R S \R S!\R S"\R S-S#\1S-S$\1S%\\"4S&jjr2"S'S(\RZ5r3"S)S*\RZ5r4"S+S,\RZ5r5"S-S.\5r6\#"S/S0\55r7\#"S1S2\755r8"S3S4\RZ5r9\#"S5S9"S6S7\755r:\#"S8S9"S9S:\755r;"S;S<\RZ5r<"S=S>\RZ5r="S?S@\RZ5r>"SASB\RZ5r?"SCSD\RZ5r@"SESF\RZ5rA"SGSH\RZ5rB\#"SISJ\755rC\#"SKS9"SLSM\\755rD/SNQrEg)P)CallableIterable) dataclassN)Tensornn)initialization)ACT2FN) BackboneMixinfilter_output_hidden_states)create_bidirectional_mask)GradientCheckpointingLayer)BackboneOutputBaseModelOutputWithPoolingImageClassifierOutputMaskedLMOutputSemanticSegmenterOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)#compile_compatible_method_lru_cache)TransformersKwargsauto_docstring torch_int)can_return_tuplemerge_with_config_defaults)capture_outputs) BeitConfigz- Class for outputs of [`BeitModel`]. ) custom_introc\rSrSrSrSrg)BeitModelOutputWithPooling0a2 pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`): Average of the last layer hidden states of the patch tokens (excluding the *[CLS]* token) if *config.use_mean_pooling* is set to True. If set to False, then the final hidden state of the *[CLS]* token will be returned. N)__name__ __module__ __qualname____firstlineno____doc____static_attributes__r$`/home/wildlama/miniconda3/lib/python3.13/site-packages/transformers/models/beit/modeling_beit.pyr"r"0sr+r"cn^\rSrSrSrS\4U4SjjrS\RS\R4Sjr Sr U=r $) BeitPatchEmbeddings?z This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. configc>[TU]5 URnURn[ U[ 5(aUOX"4n[ U[ 5(aUOX34nUSUS-USUS--UlX lX0lURUl[R"URURX3S9Ul g)Nrr kernel_sizestride) super__init__ image_size patch_size isinstancer num_patches num_channelsrConv2d hidden_size projection)selfr0r7r8 __class__s r,r6BeitPatchEmbeddings.__init__Fs && && #-j(#C#CZ*Ia #-j(#C#CZ*Ia &qMZ]:z!}PZ[\P]?]^$$"//))F$7$79K9KYcwr+ pixel_valuesreturncURSnX R:wa[SURSUS35eURU5R S5R SS5$)NrzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .)shaper; ValueErrorr>flatten transpose)r?rBr;s r,forwardBeitPatchEmbeddings.forwardSss#))!, ,, ,!../yaI |,44Q7AA!QGGr+)r7r;r:r8r>) r%r&r'r(r)rr6torchrrKr* __classcell__r@s@r,r.r.?s: xz xHELLHU\\HHr+r.c^\rSrSrSrS\SS4U4SjjrS\RS\ S \ S\R4S jr SS \RS \RS-S\R4S jjr Sr U=r$)BeitEmbeddings]zZ Construct the CLS token, position and patch embeddings. Optionally, also the mask token. r0rCNc>[TU]5 [R"[R "SSUR 55UlUR(a6[R"[R "SSUR 55OSUl [U5Ul URUl URRnUR(a9[R"[R "SUS-UR 55OSUl[R "UR"5Ulg)Nr)r5r6r ParameterrMzerosr= cls_tokenuse_mask_token mask_tokenr.patch_embeddingsr8r: use_absolute_position_embeddingsposition_embeddingsDropouthidden_dropout_probdropout)r?r0r:r@s r,r6BeitEmbeddings.__init__bs ekk!Q8J8J&KLQWQfQf",,u{{1a9K9K'LMlp 3F ; ++++77 66 LLQ a9K9KL M  zz&"<"<= r+ embeddingsheightwidthcURSS- nURRSS- n[RR 5(dXE:XaX#:Xa UR$URSS2SS24nURSS2SS24nURSnX R -n X0R -n [ US-5n URSXU5nURSSSS5n[RRUX4SS S 9nURSSSS5RSSU5n[R"Xg4SS 9$) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNg?rrrFbicubicFsizemode align_cornersdim)rGr[rMjit is_tracingr8rreshapepermuter functional interpolateviewcat) r?r`rarbr: num_positionsclass_pos_embedpatch_pos_embedrk new_height new_widthsqrt_num_positionss r,interpolate_pos_encoding'BeitEmbeddings.interpolate_pos_encodingqsS!&&q)A- 0066q9A= yy##%%+*F6?++ +221bqb59221ab59r". __, &}c'9:)11!5G]`a)11!Q1=--33 ( 4 *11!Q1=BB1b#Nyy/;CCr+rBbool_masked_poscURu p4nURU5nUR5upxnUbIURR XxS5n UR S5R U 5n USU - -X--nURR USS5n [R"X4SS9nURbX`RXdU5-nURU5nU$Nrdrrj) rGrYrgrXexpand unsqueezetype_asrVrMrsr[rzr^) r?rBr|_rarbr` batch_sizeseq_len mask_tokensmask cls_tokenss r,rKBeitEmbeddings.forwards +001e**<8 !+!2 Q  &//00bIK",,R088ED#q4x0;3EEJ^^**:r2> YY 7Q?  # # /#&C&CJX]&^^J\\*- r+)rVr^rXrYr8r[N)r%r&r'r(r)rr6rMrintrz BoolTensorrKr*rNrOs@r,rQrQ]s >z >d >&D5<<&D&DUX&D]b]i]i&DV48ll))D0  r+rQc^\rSrSrS\SS4U4Sjjr\\"SS9S\\ \ 4S\ R4S j55r S S \ S\ R4S jjrS rU=r$)BeitRelativePositionBiasr0rCNc>[TU]5 URn[U[[ 45(dX"4nUSUR -USUR -4UlSURS-S- SURS-S- -S-Ul[R"[R"URUR55UlgNrrrFr)r5r6r7r9tuplelistr8 window_sizenum_relative_distancerrTrMrUnum_attention_headsrelative_position_bias_table)r?r0r7r@s r,r6!BeitRelativePositionBias.__init__s && *udm44$1J&qMV->->> 1 QWQbQb@bc&'$*:*:1*=&=&Aa$JZJZ[\J]F]`aFa%bef%f",.LL KK22F4N4N O- )r+ )maxsizerc SUS-S- SUS-S- -S-nUSUS-n[R"[R"[R"[R"US5[R"US5SS95SS9nUSS2SS2S4USS2SSS24- R SSS5R 5nUSS2SS2S4==USS- - ss'USS2SS2S4==USS- - ss'USS2SS2S4==SUS-S- -ss'[R"US-4S-URS 9nURS 5USS2SS24'US- USSS24'US- USS2S4'US- US 'U$) z This method creates the relative position index, modified to support arbitrary window sizes, as introduced in [MiDaS v3.1](https://huggingface.co/papers/2307.14460). rFrrrij)indexing) start_dimN)rgdtyperd)rr) rMrIstackmeshgridarangero contiguousrUrsum)rr window_areacoords_flattenrelative_coordsrelative_position_indexs r, generate_relative_position_index9BeitRelativePositionBias.generate_relative_position_indexs"#[^!3a!7A A)B12&)>)BA&(=(A%&&r+rzcSURS-S- nSURS-S- nSUS-S- nSUS-S- nURnURn Xg-S-n USU S- n U RSXTS5R SSSS5n [ R RU [U5[U54SS9n U R SSSS5RU S- S5n [R"XU S- S/5n URU5nXRS5nURUSUS-S-USUS-S-S5nUR SSS5R5nU(a?[ R RURS5X34SS S 9RS5nURS5$) ze Modification of timm.models.beit.py: Attention._get_rel_pos_bias to support arbitrary window sizes. rFrrrNrdbilinear)rgrhFrf)rrrrnrorrprqrrMrsrrrrrsqueeze)r?rrzdim_size old_height old_widthrwrx old_relative_position_bias_tableold_num_relative_distancenew_num_relative_distance old_sub_table new_sub_table new_relative_position_bias_tablerrelative_position_biass r,rK BeitRelativePositionBias.forwards!))!,,q0 ((++a/ Q'!+  A&* +/+L+L($($>$>!$.$:Q$>!89X;TWX;XY %--aKSSTUWXZ[]^_  11 :!6 )8L MT^2 &--aAq9AAB[^_B_acd +099 =VYZ=Z=\] ^, (#'"G"G "T!AB^B^_aBb!c"8!%>&003)# &?& gaj #&//22r+)rrr)FN)r%r&r'r(rr6 staticmethodrrrrMrrboolrKr*rNrOs@r,rrsp  z  d  (4'eCHo'%,,'5'4-3T-3]b]i]i-3-3r+rmodulequerykeyvalueattention_maskscalingr^kwargsc UcURS5S-n[R"XRSS55U-nUbX-n[R R US[RS9RUR5n[R RXURS9n[R"X5n U RSS5R5n X4$)NrdrFr)rkr)ptrainingr) rgrMmatmulrJrrpsoftmaxfloat32torr^rr) rrrrrrr^r attn_weights attn_outputs r,eager_attention_forwardr s**R.D(<<}}Q':;gEL!#4 ==((2U]](SVVW\WbWbcL==((6??([L,,|3K''1-88:K  $$r+c ^\rSrSrS\4U4SjjrS S\RS\RS-S\\ S\ \R\R44S jjr S r U=r $) BeitAttentioni(r0c>[TU]5 XlURUl[ USUR UR-5UlURUlUR S-Ul SUl [R"UR URUR -5Ul [R"UR URUR -SS9Ul[R"UR URUR -5Ul[R"URUR -UR 5Ulg)Nhead_dimrF)bias)r5r6r0rgetattrr=rattention_probs_dropout_probattention_dropoutr is_causalrLinearq_projk_projv_projo_projr?r0r@s r,r6BeitAttention.__init__)s  #)#=#=  F4F4F&JdJd4de !'!D!D}}d* ii 2 2F4N4NQUQ^Q^4^_ ii 2 2F4N4NQUQ^Q^4^ejk ii 2 2F4N4NQUQ^Q^4^_ ii : :T]] JFL^L^_ r+N hidden_statesrrrCc URSSn/UQSPURP7nURU5RU5R SS5nUR U5RU5R SS5nUR U5RU5R SS5n[R"URR[5n U "UUUUU4UR(dSO URURS.UD6upU R"/UQSP76R!5n UR#U 5n X4$)NrdrrF)r^r)rGrrrrrJrrr get_interfacer0_attn_implementationrrrrrnrr) r?rrr input_shape hidden_shape query_states key_states value_statesattention_interfacerrs r,rKBeitAttention.forward6sE $))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST (?(M(M KK , ,.E) %8      % $}}C$2H2HLL %  % ! "));;;;FFH kk+. ((r+) rr0rrrrrrrrr)r%r&r'r(rr6rMrrrrrKr*rNrOs@r,rr(sk `z ` /3)||) t+)+, ) u||U\\) * ))r+rcj^\rSrSrS\4U4SjjrS\RS\R4SjrSr U=r $)BeitMLPiXr0c>[TU]5 Xl[URUl[ R"URUR5Ul [ R"URUR5Ul gr) r5r6r0r hidden_act activation_fnrrr=intermediate_sizefc1fc2rs r,r6BeitMLP.__init__Ysb  #F$5$5699V//1I1IJ99V55v7I7IJr+rrCclURU5nURU5nURU5nU$r)rrrr?rs r,rKBeitMLP.forward`s4/ **=9 / r+)rr0rr r%r&r'r(rr6rMrrKr*rNrOs@r,rrXs1KzKU\\ellr+rc^\rSrSrSrS S\SS4U4SjjjrS\RS\R4Sjr S\ 4S jr S r U=r $) BeitDropPathihzStochastic depth (DropPath) per sample, for residual blocks. Identity when ``drop_prob`` is 0 or outside training. See `Deep Networks with Stochastic Depth `_. drop_probrCNc.>[TU]5 Xlgr)r5r6r)r?rr@s r,r6BeitDropPath.__init__os "r+rcVURS:XdUR(dU$SUR- nURS4SURS- --n[R "X1R URS9n[R"XB-5nURU5U-$)Nrrr)r)rdevice) rrrGndimrMrandrrfloordiv)r?r keep_probrG random_tensors r,rKBeitDropPath.forwardss >>S   & $$Q')DM4F4F4J,KK 50C0CML`L`a  M$=>   +m;;r+c SUR3$)Nzp=rr?s r, extra_reprBeitDropPath.extra_repr|sDNN#$$r+rr)r%r&r'r(r)floatr6rMrrKstrrr*rNrOs@r,rrhsL #%#$##[TU]5 [U5Ul[R "UR URS9Ul[R "UR URS9Ul [U5Ul [R"UR5UlURUlUS:a [!U5O[R""5UlUR&nUS:a6[R("U[*R,"UR 5-SS9OSUlUS:a6[R("U[*R,"UR 5-SS9OSUlUR2(a[5U5UlgSUlg)NepsrrT) requires_gradg?)r5r6r attentionr LayerNormr=layer_norm_epslayernorm_beforelayernorm_afterrmlpr\r]r^r8rIdentity drop_pathlayer_scale_init_valuerTrMoneslambda_1lambda_2use_relative_position_biasrr)r?r0r  init_valuesr@s r,r6BeitLayer.__init__s@ &v. " V-?-?VEZEZ [!||F,>,>FDYDYZ6?zz&"<"<=  ++9G#9Mn5SUS^S^S`33 ^ilm^mBLLuzz&2D2D'EEUY Zsv _jlm^mBLLuzz&2D2D'EEUY Zsv KQJkJk&>v&F#qu#r+Nrrrz resolutionrrCc URbFUupgX`R-XpR-4nURXURSS9n UbX-OU nUn URU5nUR"U4SU0UD6upUR U5nUR U-nURU5U -nUn URU5nURU5nUR U5nURU-nURU5U -nU$)Nr)rr) rr8rGrrr^rrrrr) r?rrrzr!rrarbrrresidualrs r,rKBeitLayer.forwards6  & & 2&MF!__4e6NOK%)%@%@ @S@STU@V&A& ">  )    ]3  5 }5@ !,,]; /  ]3  5 }5@ r+) rrr^rrrrrr8rrNFN)r%r&r'r(r)rr r6rMrrrrrrrKr*rNrOs@r,r r sIvzv5vv,/3).-1 &||& t+&#' & #s(Od* & +, & &&r+r c^\rSrSr%\\S'SrSrSrSr S/r Sr Sr Sr SrSr\\S .rS rS /r\R,"5U4S j5rS rU=r$)BeitPreTrainedModelir0beitrB)imageTr F)r attentionsrYz.*relative_position_index.*c>[TU]U5 [U[5(a|[R "UR 5 URb [R "UR5 URb![R "UR5 gg[U[5(a![R "UR5 g[U[5(a[UR[R5(ak[R"URUR R"5 [R"UR$UR R"5 ggg)zInitialize the weightsN)r5 _init_weightsr9rQinitzeros_rVrXr[rrr rrrT constant_r0rr)r?rr@s r,r,!BeitPreTrainedModel._init_weightss f% fn - - KK(( )  , F--.))5 F6676  8 9 9 KK;; <  * *&//2<<88v 0R0RSv 0R0RS9+r+r$)r%r&r'r(r__annotations__base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backend_can_compile_fullgraphr r_can_record_outputs_input_embed_layer"_keys_to_ignore_on_load_unexpectedrMno_gradr,r*rNrOs@r,r'r's~ $O!&*#$ N "&!"#,*H)I& ]]_TTr+r'c^\rSrSrSS\S\SS4U4Sjjjr\\"SS9\ SS \ RS \ RS-S \S \ RS-S \ \S\4 Sjj555rSrU=r$) BeitModelir0add_pooling_layerrCNc >[TU]U5 Xl[U5UlUR (a [ U5OSUl[UR5Vs/sH+o1RU-[URS- S5- PM- nn[R"UVs/sH n[XS9PM sn5UlUR (a[R""5O([R$"UR&UR(S9UlU(a [-U5OSUlUR15 gs snfs snf)z^ add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer Nr)r r)r5r6r0rQr`!use_shared_relative_position_biasrshared_position_biasrangenum_hidden_layersr maxr ModuleListr layersuse_mean_poolingrrr=r layernorm BeitPoolerpooler post_init)r?r0rBidrop_path_ratesrr@s r,r6BeitModel.__init__s   (0060X0X $V ,^b !W\\b\t\tVu VuQR ! !A %F,D,Dq,H!(L LVu  mmRa$bRaQYv%HRa$bc $44BKKM",,vGYGY_e_t_t:u ->j(4   %cs !2E)E F)tie_last_hidden_statesrBr|rzrrc URXS9nURSSn[URUUS9nURbZUupXRR -XRR -4n UR XURSS9n UbX-OU nUn UR Hn U "U 4UUUS.UD6n M URU 5nURbURU5OSn[XS9$) z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). )r|rFN)r0 inputs_embedsrr)rzr)rrzr!)last_hidden_state pooler_output) r`rGr r0rEr8rJrLrNr")r?rBr|rzrrembedding_outputr!rarbrshared_relative_position_biasrlayersequence_output pooled_outputs r,rKBeitModel.forwards' ??2  ) [[E!-)A%   M!..78< 8O O4UY )Oiir+)r0r`rLrJrNrE)Tr%)r%r&r'r(rrr6rrrrMrrrrr"rKr*rNrOs@r,rArAszdd2 E248)..2 -jll-j))D0-j#' -j  t+ -j +, -j $-j3 -jr+rAcn^\rSrSrS\SS4U4SjjrS\RS\R4SjrSr U=r $) rMi3r0rCNc>[TU]5 UR(a/[R"UR UR S9UlgSUlg)Nr)r5r6rKrrr=rrLrs r,r6BeitPooler.__init__4sA KQKbKbBLL++1F1F G hl r+rcURb,URUSS2SS2SS24RS55$USS2S4$)Nrr)rLmeanrs r,rKBeitPooler.forward:sDBF..B\t~~mAqr1H5::1=>ubopqstptbuur+)rLrrOs@r,rMrM3s: z d vU\\vellvvr+rMa Beit Model transformer with a 'language' modeling head on top. BEiT does masked image modeling by predicting visual tokens of a Vector-Quantize Variational Autoencoder (VQ-VAE), whereas other vision models like ViT and DeiT predict RGB pixel values. As a result, this class is incompatible with [`AutoModelForMaskedImageModeling`], so you will need to use [`BeitForMaskedImageModeling`] directly if you wish to do masked image modeling with BEiT. c^\rSrSrS\SS4U4SjjrSr\\SS\ RS-S\ RS-S \ RS-S \ S \ RS-S \ \S\\-4S jj55rSrU=r$)BeitForMaskedImageModelingi?r0rCNc@>[TU]U5 URUl[USS9Ul[ R "URURS9Ul [ R"URUR5Ul UR5 g)NFrBr)r5r6 num_labelsrAr(rrr=rrLr vocab_sizelm_headrOrs r,r6#BeitForMaskedImageModeling.__init__Hsu   ++f> f&8&8f>S>STyy!3!3V5F5FG  r+cgrr$rs r,get_output_embeddings0BeitForMaskedImageModeling.get_output_embeddingsUsr+rBr|labelsrzrrc (UR"U4UUUS.UD6nURnURU5nURUSS2SS245n Sn Ub [R "5n U "XU5n [ U U URURS9$)a bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, BeitForMaskedImageModeling >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("microsoft/beit-base-patch16-224-pt22k") >>> model = BeitForMaskedImageModeling.from_pretrained("microsoft/beit-base-patch16-224-pt22k") >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, logits = outputs.loss, outputs.logits >>> list(logits.shape) [1, 196, 8192] ```)r|rzrNrlosslogitsrr*) r(rWrLrkrCrossEntropyLossrrr*) r?rBr|rprzrroutputsr\prediction_scoresmasked_lm_lossloss_fcts r,rK"BeitForMaskedImageModeling.forwardXsX))  +%=)    "33..9 LLAB)?@  **,H%&7&H&QN$!//))   r+)r(rLrkri)NNNFN)r%r&r'r(rr6rnrrrMrrrrrrrrKr*rNrOs@r,rfrf?s z d -137&*)..2 @ llT)@ ))D0@  t# @ #' @  t+ @ +,@   @ @ r+rfz Beit Model transformer with an image classification head on top (a linear layer on top of the average of the final hidden states of the patch tokens) e.g. for ImageNet. c^\rSrSrS\SS4U4Sjjr\\S S\RS-S\RS-S\ S \ \ S\ \-4 S jj55rS rU=r$) BeitForImageClassificationir0rCNc.>[TU]U5 URUl[USS9UlURS:a+[ R "URUR5O[ R"5Ul UR5 g)NTrhr) r5r6rirAr(rrr=r classifierrOrs r,r6#BeitForImageClassification.__init__ss   ++f= OUN_N_bcNc"))F$6$68I8IJikititiv r+rBrprzrc UR"U4SU0UD6nURnURU5nSnUbURX'UR5n[ UUUR URS9$)ab labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). rzNrr)r(rXr~ loss_functionr0rrr*) r?rBrprzrrvr]rtrss r,rK"BeitForImageClassification.forwards))  %=    -- /  %%fdkkBD$!//))   r+)r(r~riNNF)r%r&r'r(rr6rrrMrrrrrrrKr*rNrOs@r,r|r|s z d -1&*).  llT)   t#  #'  +,  & &    r+r|c^\rSrSrSS\S\S\\\\4-S\S\\\\4-\-S\S\\\\4-S \S \4U4S jjjrS \ RS \ R4Sjr Sr U=r $) BeitConvLayeri in_channels out_channelsr3r4paddingrdilationgroups activationc >[T U]5 [R"UUUUUUUUS9Ul[R "U5UlU b[U Ul g[R"5Ul g)N)rrr3r4rrrr) r5r6rr< convolution BatchNorm2d normalizationr rr) r?rrr3r4rrrrrr@s r,r6BeitConvLayer.__init__si 99#%#   ^^L90:0F&,BKKMr+rrCclURU5nURU5nURU5nU$r)rrrrs r,rKBeitConvLayer.forwards6((7 **=9  6 r+)rrr)rrrFrrrelu)r%r&r'r(rrr rr6rMrrKr*rNrOs@r,rrs .//0*+ ZZZ5c?* Z  Z uS#X&, ZZc3h'ZZZZ4U\\ellr+rc^\rSrSrS\S\S\SS4U4SjjrS\RS \\\4S\R4S jr S r U=r $) BeitPyramidPoolingBlocki pool_scalerchannelsrCNcv>[TU]5 [R"U5Ul[ X#SS9Ulg)Nrr3)r5r6rAdaptiveAvgPool2dpoolingrconv)r?rrrr@s r,r6 BeitPyramidPoolingBlock.__init__s. ++J7 !+QG r+inputrgcURU5nURU5n[RR X2SSS9nU$)NrFrf)rrrrprq)r?rrg hidden_states r,rKBeitPyramidPoolingBlock.forwards@||E* yy. }}00zin0o r+)rr) r%r&r'r(rr6rMrrrKr*rNrOs@r,rrsXH3HSHCHDH U\\sCxU\\r+rc^\rSrSrSrS\\S4S\S\SS4U4S jjrS \RS\ \R4S jr S r U=r $) BeitPyramidPoolingModuleiaK Pyramid Pooling Module (PPM) used in PSPNet. Args: pool_scales (tuple[int]): Pooling scales used in Pooling Pyramid Module. in_channels (int): Input channels. channels (int): Channels after modules, before conv_seg. Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. pool_scales.rrrCNc >[TU]5 XlX lX0l[ R "UVs/sH n[XBUS9PM sn5Ulgs snf)N)rrr) r5r6rrrrrIrblocks)r?rrrrr@s r,r6!BeitPyramidPoolingModule.__init__sY && mm#. "-J(:aij"-    sArcrUR5SSnURVs/sHo3"XS9PM sn$s snf)NrF)rg)rgr)r?r original_sizeblocks r,rK BeitPyramidPoolingModule.forwards8%**,QR0 FJkkRkUm8kRRRs4)rrrr)r%r&r'r(r)rrr6rMrrrKr*rNrOs@r,rrs\   E#s(O  #  QT  Y]  SU\\Sd5<<6HSSr+rc^\rSrSrSrS\SS4U4SjjrS\\RS\R4Sjr S \\RS\R4S jr S r U=r $) BeitUperHeadi"z Unified Perceptual Parsing for Scene Understanding. This head is the implementation of [UPerNet](https://huggingface.co/papers/1807.10221). Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. r0rCNc >[TU]5 URUlUR/S-UlURUl[ R"UR URSS9Ul [URURSUR 5Ul [URS[UR5UR --UR SSS9Ul[ R"5Ul[ R"5UlURSSHinUR R%[X R SS95 UR"R%[UR UR SSS95 Mk [[UR5UR -UR SSS9Ulg)Nrrrdrr3r)r5r6rr=rrrr<rir~r psp_modulesrlenpsp_bottleneckrI lateral_convs fpn_convsappendfpn_bottleneck)r?r0rr@s r,r6BeitUperHead.__init__*s| !--"../!3** ))DMM63D3DRST4      R MM  ,   R 3t'7'7#84==#H H MM   ]]_++CR0K    % %mK\]&^ _ NN ! !- t}}Z[ef"g h1,   !DMM 1 MM  r+rc|USn[R"U/URU5QSS9nURU5$r~)rMrsrr)r?rrs r, psp_forwardBeitUperHead.psp_forwardLsA$R( yy,!P1A1A,1O!PVWX ""<00r+encoder_hidden_statesc &/n[URU5Hup4URU"U55 M URURU55 [ U5n[ US- SS5HFnX&S- R SSnX&S- [RRX&USSS9-X&S- 'MH /n[ US- 5H)nURURU"X&55 M+ URUS5 [ US- SS5H7n[RRXUSR SSSSS9X'M9 [R"USS9nURU5n URU 5n U $) NrrrdrFrFrfrj)ziprrrrrFrGrrprqrrMrsrr~) r?rlaterals lateral_convrused_backbone_levelsrP prev_shapefpn_outsoutputs r,rKBeitUperHead.forwardQs*-d.@.@BW*X &L OOL6 7+Y (()>?@ #8}+a/B7A!a%..qr2J&1uo 0I0I *:U1J1HUO8+a/0A OODNN1-hk: ;1  %+a/B7A--33 (1+"3"3AB"7jX]4HK899X1-$$X.( r+) rr~rrrrrrr)r%r&r'r(r)rr6rrMrrrKr*rNrOs@r,rr"sa  z  d  D1ell);1 1 T%,,-?ELLr+rc ^\rSrSrSrS S\S\S\S\\\\4-SS4 U4S jjjrS \ \ RS\ R4S jr S r U=r$) BeitFCNHeadisa Fully Convolution Networks for Semantic Segmentation. This head is implemented of [FCNNet](https://huggingface.co/papers/1411.4038>). Args: config (BeitConfig): Configuration. in_channels kernel_size (int): The kernel size for convs in the head. Default: 3. dilation (int): The dilation rate for convs in the head. Default: 1. Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. r0in_indexr3rrCNc &>[TU]5 URUlURUlUR UlURUl X l US-U-n[R"5Ul URS:aURR[URUR X5US95 [!URS- 5H=nURR[UR UR UUUS95 M? UR(a4[URUR -UR X3S-S9Ul[R$"UR UR&SS9Ulg)NrFr)r3rrrrr)r5r6r=rauxiliary_channelsrauxiliary_num_convs num_convsauxiliary_concat_input concat_inputrrrIconvsrrrFconv_catr<rir~)r?r0rr3r conv_paddingrr@s r,r6BeitFCNHead.__init__sD !--11 33"99 #q(H4 ]]_ >>A  JJ  $$dmmmu  4>>A-. !!!  $/ ,!) /   )  4==0$--[qrbrDM))DMM63D3DRSTr+rcXRnUnURH nU"U5nM UR(a%UR[R "X#/SS95nUR U5nU$)Nrrj)rrrrrMrsr~)r?rr#rrs r,rKBeitFCNHead.forwardse(7 JJD /M    MM%))X4MST*UVM 6 r+)rr~rrrrrr)rFrr)r%r&r'r(r)rrrr6rrMrrKr*rNrOs@r,rrss no!U !U,/!UBE!UUX[`adfiai[jUj!U !U!UFT%,,-?ELLr+rc ~^\rSrSrSrS S\S\S\SS4U4SjjjrS \RS\R4S jr S r U=r $) BeitFPNUpBlockiuE4x upsampling block: ConvTranspose → BN → GELU → ConvTranspose.r=r3r4rCNc>[TU]5 [R"XX#S9Ul[R "U5Ul[R"5Ul[R"XX#S9Ul g)Nr2) r5r6rConvTranspose2dconv_transpose1rrGELUrconv_transpose2)r?r=r3r4r@s r,r6BeitFPNUpBlock.__init__sX !11+Xcs^^K8'')!11+Xcsr+rcURU5nURU5nURU5nURU5nU$r)rrrrrs r,rKBeitFPNUpBlock.forwardsF,,]; **=9  6 ,,]; r+)rrrr)rFrF) r%r&r'r(r)rr6rMrrKr*rNrOs@r,rrsSOtCtctstSWttU\\ellr+rc^\rSrSrSrS\4U4SjjrS\\RS4S\\RS44Sjr S r U=r $) BeitFPNNeckiz 4-level feature pyramid neck for BeiT. Produces x4 upsample, x2 upsample, identity, and x2 downsample outputs from the four selected ViT feature maps. r0c>[TU]5 [UR5Ul[ R "URURSSS9Ul[ R"SSS9Ul g)NrFr2) r5r6rr=fpn1rrfpn2 MaxPool2dfpn4rs r,r6BeitFPNNeck.__init__sX "6#5#56 &&v'9'96;M;M[\efg LLQq9 r+ feature_maps.rCcURUS5URUS5USURUS54$rrrr)r?rs r,rKBeitFPNNeck.forwardsC IIl1o & IIl1o & O IIl1o &   r+r) r%r&r'r(r)rr6rrMrrKr*rNrOs@r,rrsI :z:  E%,,*;$< u||UXGXAY  r+rc^\rSrSrS\SS4U4Sjjr\\\S S\ RS-S\ RS-S\ S \ \ S\\-4 S jj555rS rU=r$) BeitForSemanticSegmentationir0rCNcf>[TU]U5 URUl[USS9Ul[ UR R5S:wa [S5e[U5Ul [U5Ul UR(a [U5OSUlUR!5 g)NFrhrzBeitForSemanticSegmentation requires config.out_indices to be a list of 4 integers, specifying which features to use from the backbone. One can use [3, 5, 7, 11] in case of a base-sized architecture.)r5r6rirAr(rr0 out_indicesrHrfpnr decode_headuse_auxiliary_headrauxiliary_headrOrs r,r6$BeitForSemanticSegmentation.__init__s   ++f> t{{&& '1 ,-  v&(/5;5N5Nk&1TX r+rBrprzrc ^ ^^^Ub%URRS:Xa [S5eUR"U4SU0UD6nURmUR um pgnXpRR -mXRR -m[U UUU4SjURR55n URU 5n URU 5n Sn URbURU 5n Sn Ub;URU UURRU URRS9n [U U URUR S9$)a labels (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*): Ground truth semantic segmentation maps for computing the loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1`, a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, BeitForSemanticSegmentation >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("microsoft/beit-base-finetuned-ade-640-640") >>> model = BeitForSemanticSegmentation.from_pretrained("microsoft/beit-base-finetuned-ade-640-640") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> # logits are of shape (batch_size, num_labels, height, width) >>> logits = outputs.logits ```Nrz/The number of labels should be greater than onerzc3># UH8nTUS- SS2SS24RSS5RTSTT5v M: g7f)rNrFrd)rJrn).0rPrr patch_height patch_widths r, 6BeitForSemanticSegmentation.forward..sP , "!a% (AB / 9 9!Q ? G G TVXdfq r r,sAA) ignore_indexauxiliary_logitsauxiliary_loss_weightrr)r0rirHr(rrGr8rrrrrrsemantic_loss_ignore_indexrrr*)r?rBrprzrrvrrarbrrtrrsrrrrs @@@@r,rK#BeitForSemanticSegmentation.forwards`B  $++"8"8A"=NO O))  %=   !( 5 5'3'9'9$ Au!7!77 {{555  [[,,  xx - !!,/    *#22<@   %%![[CC!1&*kk&G&G &D'!//))   r+)rr(rrrir)r%r&r'r(rr6rr rrMrrrrrrrKr*rNrOs@r,rrszd* -1&*). G llT)G  t#G #' G +, G ( ( G !G r+rzM BEiT backbone, to be used with frameworks like DETR and MaskFormer. c b^\rSrSrU4Sjr\\\S\S\ \ S\ 4Sj555r Sr U=r$) BeitBackbonei;cF>[TU]U5 [URS-5Vs/sHo!RPM snUl[ USS9UlUR(a [U5O[R"5Ul UR5 gs snf)NrFrh)r5r6rFrGr= num_featuresrAr(add_fpnrrrrrO)r?r0rr@s r,r6BeitBackbone.__init__Asx  9>v?W?WZ[?[9\]9\A//9\]f> *0..;v&bkkm  ^sBrBrrCc (URup4pVXPRR-nX`RR-nUR"U40UD6n U Rn Sn [ UR U 5HiupXR;dMURR(a3U SS2SS2SS24n U RSS5n U RUSXx5n X4- n Mk URU 5n [U U RU RS9$)a Examples: ```python >>> from transformers import AutoImageProcessor, AutoBackbone >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> processor = AutoImageProcessor.from_pretrained("microsoft/beit-base-patch16-224") >>> model = AutoBackbone.from_pretrained( ... "microsoft/beit-base-patch16-224", out_features=["stage1", "stage2", "stage3", "stage4"] ... ) >>> inputs = processor(image, return_tensors="pt") >>> outputs = model(**inputs) >>> feature_maps = outputs.feature_maps >>> list(feature_maps[-1].shape) [1, 768, 14, 14] ```r$NrrFrd)rrr*)rGr0r8r(rr stage_names out_featuresreshape_hidden_statesrJrnrrr*)r?rBrrrrarbrrrvrrstagers r,rKBeitBackbone.forwardKs@(4'9'9$ v!7!77 {{555 ))L3F3--  #&t'7'7#G E)));;44#/12q#9L#/#9#9!Q#?L#/#7#7 B #bL/ $Hxx - %!//))  r+)r(rr)r%r&r'r(r6rr rrrrrrKr*rNrOs@r,rr;sN  3 3 +,3   3 !3 r+r)r|rfrrAr'r)Nr)Fcollections.abcrr dataclassesrrMrrr r- activationsr backbone_utilsr r masking_utilsr modeling_layersrmodeling_outputsrrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsrrr utils.genericrrutils.output_capturingrconfiguration_beitrr"Moduler.rQrr rrrrr r'rArMrfr|rrrrrrrrr__all__r$r+r,rs*/! &!H69G&@BBI5*  !;  H"))H