3jdnVSSKJrJr SSKJr SSKrSSKJr SSKJr SSK J r SSK J r SS KJr SS KJrJrJr SS KJrJr SS KJr SS KJrJrJrJr SSKJrJ r SSK!J"r" SSK#J$r$ "SS\RJ5r&"SS\RJ5r'S9S\RJS\RPS\RPS\RPS\RPS-S\)S-S\)S\\4Sjjr*"SS \RJ5r+"S!S"\RJ5r,"S#S$\5r-\"S%S&\55r."S'S(\RJ5r/\"S)S*\.55r0\"S+S,9"S-S.\.55r1\"S/S,9"S0S1\.55r2\"S2S,9\"S3S4\555r3\"S5S,9"S6S7\.55r4/S8Qr5g):)CallableIterable) dataclassN)nn)initialization)ACT2FN)create_bidirectional_mask)GradientCheckpointingLayer)BaseModelOutputWithPoolingImageClassifierOutputMaskedImageModelingOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstring torch_int)can_return_tuplemerge_with_config_defaults)capture_outputs) DeiTConfigcn^\rSrSrSrS\4U4SjjrS\RS\R4Sjr Sr U=r $) DeiTPatchEmbeddings(z This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. configc>[TU]5 URnURn[ U[ 5(aUOX"4n[ U[ 5(aUOX34nUSUS-USUS--UlX lX0lURUl[R"URURX3S9Ul g)Nrr) kernel_sizestride) super__init__ image_size patch_size isinstancer num_patches num_channelsrConv2d hidden_size projection)selfrr$r% __class__s `/home/wildlama/miniconda3/lib/python3.13/site-packages/transformers/models/deit/modeling_deit.pyr#DeiTPatchEmbeddings.__init__/s && && #-j(#C#CZ*Ia #-j(#C#CZ*Ia &qMZ]:z!}PZ[\P]?]^$$"//))F$7$79K9KYcw pixel_valuesreturncURSnX R:wa[SURSUS35eURU5R S5R SS5$)NrzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .)shaper( ValueErrorr+flatten transpose)r,r1r(s r.forwardDeiTPatchEmbeddings.forward<ss#))!, ,, ,!../yaI |,44Q7AA!QGGr0)r$r(r'r%r+) __name__ __module__ __qualname____firstlineno____doc__rr#torchTensorr:__static_attributes__ __classcell__r-s@r.rr(s: xz xHELLHU\\HHr0rc ^\rSrSrSrSS\S\SS4U4SjjjrS\RS \ S \ S\R4S jr SS \RS \RS-S\S\R4Sjjr SrU=r$)DeiTEmbeddingsFa Construct the CLS token, distillation token, position and patch embeddings. Optionally, also the mask token. Differences from ViTEmbeddings: - Adds a distillation token (for distillation pre-training). - Position embeddings include +2 slots (CLS + distillation) instead of +1. - interpolate_pos_encoding handles 2 special tokens instead of 1. - forward concatenates distillation token and handles position encoding for both. ruse_mask_tokenr2Nc>[TU]5 [R"[R "SSUR 55UlU(a6[R"[R "SSUR 55OSUl[U5Ul URRn[R"[R "SUS-UR 55Ul [R"UR5UlUR UlURR"Ul[R"[R "SSUR 55Ulg)Nrr5)r"r#r ParameterrAzerosr* cls_token mask_tokenrpatch_embeddingsr'position_embeddingsDropouthidden_dropout_probdropoutr%r$distillation_token)r,rrIr'r-s r.r#DeiTEmbeddings.__init__Qs ekk!Q8J8J&KLQ_",,u{{1a9K9K'LMei 3F ;++77 #%<< A{QPVPbPb0c#d zz&"<"<=  ++//::"$,,u{{1aASAS/T"Ur0 embeddingsheightwidthcURSS- nURRSS- n[RR 5(dXE:XaX#:Xa UR$URSS2SS24nURSS2SS24nURSnX R -n X0R -n [ US-5n URSXU5nURSSSS5n[RRUX4SS S 9nURSSSS5RSSU5n[R"Xg4SS 9$) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing and 2 class embeddings. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rr5N?rrbicubicF)sizemode align_cornersdim)r6rPrAjit is_tracingr%rreshapepermuter functional interpolateviewcat) r,rVrWrXr' num_positionsclass_and_dist_pos_embedpatch_pos_embedra new_height new_widthsqrt_num_positionss r.interpolate_pos_encoding'DeiTEmbeddings.interpolate_pos_encoding^sU!&&q)A- 0066q9A= yy##%%+*F6?++ +#'#;#;ArrE#B 221ab59r". __, &}c'9:)11!5G]`a)11!Q1=--33 ( 4 *11!Q1=BB1b#Nyy2D!LLr0r1bool_masked_posrpc URu pEnURU5nUR5upnUbIURR XS5n UR S5R U 5n USU - -X--nURR USS5n URR USS5n [R"XU4SS9nU(aXpRXuU5-nOdXPRS:wdX`RS:wa2[SUSUSURSSURSS 3 5eXpR-nURU5nU$) NrZg?rr`rzInput image size (*z) doesn't match model (z).)r6rOr]rNexpand unsqueezetype_asrMrTrArirpr$r7rPrS)r,r1rrrp_rWrXrV batch_size seq_length mask_tokensmask cls_tokensdistillation_tokenss r.r:DeiTEmbeddings.forwardsm +001e**<8 $.OO$5!   &//00LK",,R088ED#sTz2[5GGJ^^**:r2> "55<&>>J\\*- r0)rMrTrSr$rNrOr%rP)F)NF)r<r=r>r?r@rboolr#rArBintrp BoolTensorr:rCrDrEs@r.rGrGFs Vz V4 VD V V&M5<<&M&MUX&M]b]i]i&MV48). ll ))D0 #'    r0rGmodulequerykeyvalueattention_maskscalingrSkwargsc UcURS5S-n[R"XRSS55U-nUbX-n[R R US[RS9RUR5n[R RXURS9n[R"X5n U RSS5R5n X4$)NrZr5r)radtype)ptrainingr) r]rAmatmulr9rrfsoftmaxfloat32torrSr contiguous) rrrrrrrSr attn_weights attn_outputs r.eager_attention_forwardrs**R.D(<<}}Q':;gEL!#4 ==((2U]](SVVW\WbWbcL==((6??([L,,|3K''1-88:K  $$r0c ^\rSrSrS\4U4SjjrS S\RS\RS-S\\ S\ \R\R44S jjr S r U=r $) DeiTAttentionrc>[TU]5 XlURUl[ USUR UR-5UlURUlUR S-Ul SUl [R"UR URUR -URS9Ul[R"UR URUR -URS9Ul[R"UR URUR -URS9Ul[R"URUR -UR SS9Ulg)Nhead_dimrF)biasT)r"r#rnum_attention_headsgetattrr*rattention_probs_dropout_probattention_dropoutr is_causalrLinearqkv_biasq_projk_projv_projo_projr,rr-s r.r#DeiTAttention.__init__s*  #)#=#=  F4F4F&JdJd4de !'!D!D}}d* ii 2 2F4N4NQUQ^Q^4^eketetu ii 2 2F4N4NQUQ^Q^4^eketetu ii 2 2F4N4NQUQ^Q^4^eketetu ii : :T]] JFL^L^eij r0N hidden_statesrrr2c URSSn/UQSPURP7nURU5RU5R SS5nUR U5RU5R SS5nUR U5RU5R SS5n[R"URR[5n U "UUUUU4UR(dSO URURS.UD6upU R"/UQSP76R!5n UR#U 5n X4$)NrZrr5)rSr)r6rrrhr9rrr get_interfacer_attn_implementationrrrrrdrr) r,rrr input_shape hidden_shape query_states key_states value_statesattention_interfacerrs r.r:DeiTAttention.forwardsE $))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST (?(M(M KK , ,.E) %8      % $}}C$2H2HLL %  % ! "));;;;FFH kk+. ((r0) rrrrrrrrrrN)r<r=r>r?rr#rArBrrtupler:rCrDrEs@r.rrsk kz k"/3)||) t+)+, ) u||U\\) * ))r0rcj^\rSrSrS\4U4SjjrS\RS\R4SjrSr U=r $)DeiTMLPrc>[TU]5 Xl[URUl[ R"URUR5Ul [ R"URUR5Ul gr) r"r#rr hidden_act activation_fnrrr*intermediate_sizefc1fc2rs r.r#DeiTMLP.__init__sb  #F$5$5699V//1I1IJ99V55v7I7IJr0rr2clURU5nURU5nURU5nU$r)rrr)r,rs r.r:DeiTMLP.forwards4/ **=9 / r0)rrrr r<r=r>r?rr#rArBr:rCrDrEs@r.rrs1KzKU\\ellr0rc ^\rSrSrS\4U4SjjrS S\RS\RS-S\\ S\R4S jjr S r U=r $) DeiTLayerircd>[TU]5 [U5Ul[R "UR URS9Ul[R "UR URS9Ul [U5Ul [R"UR5Ulg)Neps)r"r#r attentionr LayerNormr*layer_norm_epslayernorm_beforelayernorm_afterrmlprQrRrSrs r.r#DeiTLayer.__init__sy &v. " V-?-?VEZEZ [!||F,>,>FDYDYZ6?zz&"<"<= r0Nrrrr2c UnURU5nUR"X40UD6upURU5nX-nUnURU5nUR U5nURU5nX-nU$r)rrrSrr)r,rrrresidualrxs r.r:DeiTLayer.forwards!--m< >>-R6R  ]3 %0 !,,]; /  ]3 %0 r0)rrSrrrr) r<r=r>r?rr#rArBrrr:rCrDrEs@r.rrs[>z>/3|| t++,   r0rc^\rSrSr%\\S'SrSrSrSr SS/r Sr Sr Sr SrSr\\S .rS r\R*"5S \R.\R0-\R2-S S 4U4Sjj5rSrU=r$)DeiTPreTrainedModeli&rdeitr1)imageTrGr)r attentionsrOrr2Nc>[TU]U5 [U[5(aURb4[ R "URSURRS9 [ R "URSURRS9 URb [ R"UR5 [U[5(a[ R"UR5 [ R"UR5 [ R"UR5 URb![ R"UR5 ggg)zInitialize the weightsNr)meanstd) r" _init_weightsr&rGrPinit trunc_normal_rinitializer_rangerMrNzeros_rT)r,rr-s r.r!DeiTPreTrainedModel._init_weights9s f% fn - -))5""6#=#=CT[[MjMjk   v//ct{{?\?\ ]  , F--. fn - - KK(( ) KK22 3 KK11 2  , F--.- .r0)r<r=r>r?r__annotations__base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backend_can_compile_fullgraphrr_can_record_outputs_input_embed_layerrAno_gradrrr)rrrCrDrEs@r.rr&s $O!&*#);7N"&!"#, ]]_/BII $9BLL$H/T//r0rcj^\rSrSrS\4U4SjjrS\RS\R4SjrSr U=r $) DeiTPooleriKrc>[TU]5 [R"URUR 5Ul[URUl gr) r"r#rrr*pooler_output_sizedenser pooler_act activationrs r.r#DeiTPooler.__init__Ls> YYv1163L3LM  !2!23r0rr2c\USS2S4nURU5nURU5nU$)Nr)rr)r,rfirst_token_tensor pooled_outputs r.r:DeiTPooler.forwardQs6+1a40 #56  6 r0)rrrrEs@r.rrKs/4z4 U\\ellr0rc^\rSrSrSS\S\S\4U4Sjjjr\\"SS9\ SS \ RS-S \ RS-S \S-S \ RS-S \ \S\4 Sjj555rSrU=r$) DeiTModeliZFradd_pooling_layerrIc>[TU]U5 Xl[XS9Ul[ R "[UR5Vs/sHn[U5PM sn5Ul [ R"URURS9UlU(a [U5OSUlUR#5 gs snf)z add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer use_mask_token (`bool`, *optional*, defaults to `False`): Whether to use a mask token for masked image modeling. )rIrN)r"r#rrGrVr ModuleListrangenum_hidden_layersrlayersrr*r layernormrpooler post_init)r,rrrIrxr-s r.r#DeiTModel.__init__\s   (OmmfF^F^@_$`@_1Yv%6@_$`a f&8&8f>S>ST,=j(4   %as C)tie_last_hidden_statesNr1rrrprrr2c URRRRRnURU:waUR U5nURXUS9n[ URUUS9nUnURH n U "X40UD6nM URU5n URbURU 5OSn [XS9$)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). )rrrp)r inputs_embedsrN)last_hidden_state pooler_output) rVrOr+weightrrr rrrrr ) r,r1rrrprrexpected_dtypeembedding_outputrlayersequence_outputrs r.r:DeiTModel.forwardls 99DDKKQQ    /'??>:L?? Tl+ 3;;*)  ) [[E!-J6JM!..78< 8O O4UY )Oiir0)rrVrrr)TFNNNN)r<r=r>r?rrr#rrrrArBrrrr r:rCrDrEs@r.rrZszd[_  E2-13704.2 jllT) j))D0 j#'+ j  t+ j +, j $ j3  jr0rad DeiT Model with a decoder on top for masked image modeling, as proposed in [SimMIM](https://huggingface.co/papers/2111.09886). Note that we provide a script to pre-train this model on custom data in our [examples directory](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining). ) custom_introc^\rSrSrS\4U4Sjjr\\S S\RS-S\RS-S\ S\RS-S \ \ S \4 S jj55rS rU=r$)DeiTForMaskedImageModelingircH>[TU]U5 [USSS9Ul[R "[R "URURS-UR-SS9[R"UR55Ul UR5 g)NFT)rrIr5r) in_channels out_channelsr ) r"r#rrr Sequentialr)r*encoder_strider( PixelShuffledecoderrrs r.r##DeiTForMaskedImageModeling.__init__s  fdS }} II"..#22A58K8KK  OOF11 2    r0Nr1rrrprrr2c RUR"U4UUUS.UD6nURnUSS2SS24nURupn [U S-5=pUR SSS5R XX5nUR U5n SnUGbURRURR-nUR SX5nURURRS5RURRS5RS5R5n[RRXSS 9nUU-R!5UR!5S -- URR"- n[%UU UR&UR(S 9$) a bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Examples: ```python >>> from transformers import AutoImageProcessor, DeiTForMaskedImageModeling >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> model = DeiTForMaskedImageModeling.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, reconstructed_pixel_values = outputs.loss, outputs.reconstruction >>> list(reconstructed_pixel_values.shape) [1, 3, 224, 224] ```)rrrprNr5r[rrrZnone) reductiongh㈵>)lossreconstructionrr)rr r6rrerdrrr$r%repeat_interleavervrrrfl1_losssumr(rrr)r,r1rrrprroutputsrrysequence_lengthr(rWrXreconstructed_pixel_valuesmasked_im_lossr]r|reconstruction_losss r.r:"DeiTForMaskedImageModeling.forwardsL/3ii / +%=) /  / "33*!QR%04C4I4I1 \_c122)11!Q:BB:]ck&*\\/%B"  &;;))T[[-C-CCD-55b$EO11$++2H2H!L""4;;#9#91=1  #%--"7"7 lr"7"s 1D8==?488:PTCTUX\XcXcXpXppN(5!//))   r0)rr)NNFN)r<r=r>r?rr#rrrArBrrrrrr:rCrDrEs@r.rrsz"-137)..2 J llT)J ))D0J #' J  t+ J +, J  #J J r0ra DeiT Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet. Note that it's possible to fine-tune DeiT on higher resolution images than the ones it has been trained on, by setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained position embeddings to the higher resolution. c^\rSrSrS\4U4Sjjr\\S S\RS-S\RS-S\ S-S\RS-S \ \ S \ 4 S jj55rS rU=r$)DeiTForImageClassificationirc.>[TU]U5 URUl[USS9UlURS:a+[ R "URUR5O[ R"5Ul UR5 gNF)rr) r"r# num_labelsrrrrr*Identity classifierrrs r.r##DeiTForImageClassification.__init__ss   ++f> OUN_N_bcNc"))F$6$68I8IJikititiv r0Nr1labelsrprrr2c UR"U4UUS.UD6nURnUSS2SSS24nURU5n Sn UbUR"X)UR40UD6n [ U U UR URS9$)ab labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). rprNr)r#logitsrr)rr r4 loss_functionrr rr) r,r1r6rprrr(rrr9r#s r.r:"DeiTForImageClassification.forwards"/3ii / %=)/  / "33'1a0 /  %%fdkkLVLD$!//))   r0)r4rr2r)r<r=r>r?rr#rrrArBrrrr r:rCrDrEs@r.r/r/s z -1&*04.2 # llT)#  t## #'+ #  t+ # +, #  # # r0r/zC Output type of [`DeiTForImageClassificationWithTeacher`]. c\rSrSr%SrSr\RS-\S'Sr \RS-\S'Sr \RS-\S'Sr \ \RS-\S'Sr \ \RS-\S'S rg) +DeiTForImageClassificationWithTeacherOutputiBaF logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores as the average of the cls_logits and distillation logits. cls_logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores of the classification head (i.e. the linear layer on top of the final hidden state of the class token). distillation_logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores of the distillation head (i.e. the linear layer on top of the final hidden state of the distillation token). Nr9 cls_logitsdistillation_logitsrrr)r<r=r>r?r@r9rA FloatTensorrr>r?rrrrCrr0r.r=r=Bs} (,FE   $++/J!!D(/48**T1859M5**+d2926Je''(4/6r0r=a DeiT Model transformer with image classification heads on top (a linear layer on top of the final hidden state of the [CLS] token and a linear layer on top of the final hidden state of the distillation token) e.g. for ImageNet. .. warning:: This model supports inference-only. Fine-tuning with distillation (i.e. with a teacher) is not yet supported. c ^\rSrSrS\SS4U4Sjjr\\S S\RS-S\ S\RS-S \ \ S\ 4 S jj55rS rU=r$) %DeiTForImageClassificationWithTeacheri[rr2Nc>[TU]U5 URUl[USS9UlURS:a+[ R "URUR5O[ R"5Ul URS:a+[ R "URUR5O[ R"5Ul UR5 gr1) r"r#r2rrrrr*r3cls_classifierdistillation_classifierrrs r.r#.DeiTForImageClassificationWithTeacher.__init__gs   ++f> AG@Q@QTU@UBIIf((&*;*; <[][f[f[h AG@Q@QTU@UBIIf((&*;*; <[][f[f[h $ r0r1rprrc UR"U4UUS.UD6nURnURUSS2SSS245nURUSS2SSS245nXx-S- n [ U UUUR UR S9$)Nr8rrr5)r9r>r?rr)rr rDrEr=rr) r,r1rprrr(rr>r?r9s r.r:-DeiTForImageClassificationWithTeacher.forwardxs/3ii / %=)/  / "33((Aq)AB "::?1aQR7;ST2a7:! 3!//))   r0)rDrrEr2)NFN)r<r=r>r?rr#rrrArBrrrr=r:rCrDrEs@r.rBrB[szd"-1)..2  llT) #'  t+  +,  5   r0rB)r/rBrrr)Nr)6collections.abcrr dataclassesrrArrr activationsr masking_utilsr modeling_layersr modeling_outputsr r rmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrrutils.output_capturingrconfiguration_deitrModulerrGrBfloatrrrrrrrrr/r=rB__all__rr0r.rYs*/! &!69llF&OOI5*H"))H<`RYY`R!% II% <<% % << % LL4' % T\ %%'(%8.)BII.)b bii  *@!//!/!/H  4j#4j4jn  ^ !4^  ^ B  2 !42  2 j  7+7  7& 0 ,?0  0 f r0