3j1:SrSSKrSSKJr SSKrSSKJr SSKJr SSKJ r SSK J r SS K JrJrJr SS KJr SS KJrJr SS KJr SS KJrJrJr SSKJr SSKJr SSK J!r!J"r"J#r# SSK$J%r% SSK&J'r' \"RP"\)5r*"SS\RV5r,"SS\RV5r-"SS\RV5r."SS\5r/\!"SS\55r0\!"SS\055r1\!"S S!9"S"S#\0\55r2\!"S$S!9"S%S&\055r3/S'Qr4g)(zPyTorch OpenAI ImageGPT model.N)Any)nn)CrossEntropyLoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentions SequenceClassifierOutputWithPast)PreTrainedModel)Conv1D)auto_docstringlogging torch_float)maybe_autocast)ImageGPTConfigcx^\rSrSrS S\\S\4U4SjjjrS\RS\R4Sjr Sr U=r $) ImageGPTLayerNorm0 hidden_sizeepsc>[TU]5 X l[R"[ R "U55UlgN)super__init__rr ParametertorchTensorweight)selfrr __class__s h/home/wildlama/miniconda3/lib/python3.13/site-packages/transformers/models/imagegpt/modeling_imagegpt.pyr#ImageGPTLayerNorm.__init__1s, ll5<< #<= tensorreturnc U[R"[R"[R"U5SSS9UR-5- nXR -nU$)NT)axiskeepdim)r%sqrtmeansquarerr')r(r-s r*forwardImageGPTLayerNorm.forward6sI%**UZZ V0D2W[%\_c_g_g%ghh++% r,)rr')gh㈵>) __name__ __module__ __qualname____firstlineno__tupleintfloatr#r%r&r6__static_attributes__ __classcell__r)s@r*rr0s?>E#J>U>> ellu||r,rc^\rSrSrSS\S-S\S-4U4SjjjrSSjrSSjrSr S r SS \ RS \ S-S \ RS-S \ RS-S\ RS-S\S-S\S-S\4SjjrSrU=r$)ImageGPTAttention=Nis_cross_attention layer_idxc >[TU]5 XlURnUR S[ R "[ R"XD4[ RS95RSSXD5SS9 URUl URUl URUR-UlURUlURUR-UR:wa&[!SURSURS35eUR"UlX lUR&UlX0lUR*UlUR$(aN[-S UR-UR5Ul[-URUR5UlO([-S UR-UR5Ul[-URUR5Ul[4R6"UR85Ul[4R6"UR<5Ulg) NbiasdtyperF) persistentz=`embed_dim` must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r) r"r#configmax_position_embeddingsregister_bufferr%trilonesboolviewr embed_dimnum_attention_heads num_headshead_dim split_size ValueErrorscale_attn_weightsrEscale_attn_by_inverse_layer_idxrFreorder_and_upcast_attnrc_attnq_attnc_projrDropout attn_pdrop attn_dropout resid_pdrop resid_dropout)r(rMrErF max_positionsr)s r*r#ImageGPTAttention.__init__>s  66    JJuzz="@ S T Y Y1m    ++33$..8 .. ==4>> )T^^ ;OPTP^P^O_`NN#2'  #)";";"406/U/U,"'-'E'E$  " " T^^!3T^^DDK @DK T^^!3T^^DDKT^^T^^< JJv'8'89ZZ(:(:;r,cF[R"XRSS55nUR(a U[ UR S5S-5- nUR (aU[URS-5- nUR(dUR S5UR S5pvURSS2SS2Xv- U2SU24n[R"UR5Rn [R"XRURS9n [R "XU 5nUbXT-n["R$"SS9"U5nUR'UR5nUR)U5n[R"XS5n X4$)Nr0?rrJdevicedim)r%matmul transposerZrsizer[r>rFrErHfinforJminr-rkwhererSoftmaxtyperb) r(querykeyvalueattention_mask attn_weights query_length key_length causal_mask mask_value attn_outputs r*_attnImageGPTAttention._attnfsP||E==R+@A  " "'+ejjn6K*LLL  / /'%0B*CCL&&',zz"~sxx|*))Aq**Cj*PR]S]R]$]^K\%7%78<r[rFrrureshaperobaddbmmrErHrqrJrrr-rsrrt RuntimeErrorrbrn)r(rvrwrxrybszrV q_seq_lendk_ k_seq_lenrz scale_factorqkr{r|r}r~rs r*_upcast_and_reordered_attn,ImageGPTAttention._upcast_and_reordered_attns2(- %  XXZ1{{3?IPUP]P]fkfrfrs   " " E%**R.1S8 8L  / / E$..1"45 5LELL--u ===Y3S]]2r5J5R5RSUWY5eq ==wwy!'')RS[ghL'// UL> &&',zz"~sxx|*L))Aq*|*Cj*PR]S]R]$]^K\%7%78< =s A9J$$ J3cvUR5SSX#4-nUR"U6nURSSSS5$)z: Splits hidden_size dim into attn_head_size and num_heads Nr0rrLrr)rprSpermuter(r-rVattn_head_size new_shapes r* _split_headsImageGPTAttention._split_headssAKKM#2&))DD i(~~aAq))r,cURSSSS5R5nUR5SSX#-4-nURU5$)zC Merges attn_head_size dim and num_attn_heads dim into hidden_size rrLrrNrh)r contiguousrprSrs r* _merge_headsImageGPTAttention._merge_headssM1a+668KKM#2&)*D)FF {{9%%r, hidden_states layer_pastryencoder_hidden_statesencoder_attention_mask use_cacheoutput_attentionsr.c USLn URupn Ub][U[5(aFURR UR 5n U (a UR nOURnOUnU (aUOUnU (Ga-[US5(d [S5eUb`W (aYURU5nWRUR RnURUR RnGOKURU5nURU5RUR SS9unnUR#U SUR$UR&5R)SS5nUR#U SUR$UR&5R)SS5nOURU5RUR SS9unnnUR#U SUR$UR&5R)SS5nUR#U SUR$UR&5R)SS5nUb@WR+UUUR 5unnU (aSURUR 'UR#XUR$UR&5R)SS5nUR,(aUR/UUUU5unnOUR1UUUU5unnUR3UUR$UR&5nUR5U5nUR7U5nUU4$)Nr^zIf class is used as cross attention, the weights `q_attn` have to be defined. Please make sure to instantiate class with `ImageGPTAttention(..., is_cross_attention=True)`.rLrlr0rT)shape isinstancer is_updatedgetrFcross_attention_cacheself_attention_cachehasattrrYr^layerskeysvaluesr]splitrXrSrVrWroupdater\rrrr_rd)r(rrryrrrrkwargsrErseq_lenrrcurr_past_key_valuescurrent_statesrvrwrxrrzs r*r6ImageGPTAttention.forwards3$>'--a  !*&9::'2266t~~F %+5+K+K(+5+J+J('1$2D.- 4** t %* M2*11$..AFF,33DNNCJJ M2![[8>>tTU>V UhhsB FPPQRTUV 3DNNDMMJTTUVXYZ $ N ; A A$//WX A Y E3((3DNNDMMBLLQPQRCJJsB FPPQRTUVE  !-44S%PJC!8< %%dnn5 3GQQRSUVW  ' '(,(G(GsTY[i(j %K(, 5#un(U %K'' T^^T]]S kk+. ((5 L((r,)rbr]r_rMrTrWrErFrVr^r\rdr[rZrX)FNr!NNNNFF)r8r9r:r;rRr=r#rrrrr%r&r r<r6r?r@rAs@r*rCrC=s&<4$;&[TU]5 URn[X5Ul[X15Ul[ URUl[R"UR5Ul gr!) r"r#rrc_fcr_ractivation_functionactrr`rcdropout)r(intermediate_sizerMrTr)s r*r#ImageGPTMLP.__init__sZ && ,8 Y: &445zz&"4"45 r,rr.cURU5nURU5nURU5nURU5nU$r!)rrr_r)r(rs r*r6ImageGPTMLP.forwards@ -0 /  M2  ]3 r,)rrr_r) r8r9r:r;r#r%r&r6r?r@rAs@r*rr s(6U\\ellr,rc^\rSrSrSU4SjjrSS\R S\S-S\R S-S\R S-S\R S-S \S-S \S-S \ 4S jjr S r U=r $) ImageGPTBlockiNc>[TU]5 URnURb UROSU-n[ X1R S9Ul[XS9Ul[ X1R S9Ul UR(a([USUS9Ul [ X1R S9Ul [XA5Ulg)NrrFT)rErF)r"r#rn_innerrlayer_norm_epsilonln_1rCattnln_2add_cross_attentioncrossattention ln_cross_attnrmlp)r(rMrFr inner_dimr)s r*r#ImageGPTBlock.__init__s (( &,nn&@FNNa+o %k7P7PQ %fB %k7P7PQ  % %"3Ft_h"iD !2;D]D]!^D y1r,rrryrrrrr.c zUn URU5nURUUUUUS9n U Sn U SSn X-nUbW[US5(d[SUS35eUn UR U5nUR UUUUUUS9n U Sn X-nXSS-n Un UR U5nURU5nX-nU4U -$)N)rryrrrrrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`)rryrrr)rrrrYrrrr)r(rrryrrrrrresidual attn_outputsroutputscross_attn_outputsfeed_forward_hidden_statess r*r6ImageGPTBlock.forward.s'! -0 yy !)/ ! #1o qr"#. ,4!122 =dVDZZ%H ..}=M!%!4!4%-&;'="3 "5" -Q/K$2M12 66G  -0 %)XXm%<" = '))r,)rrrrrrr!r) r8r9r:r;r#r%r&r rRr<r6r?r@rAs@r*rrs 2$$(.2596:!&).5*||5*DL5* t+ 5* %||d2 5* !& t 3 5*$;5* $;5* 5*5*r,rcr^\rSrSr%\\S'SrSrSrSr S/r \ R"5U4Sj5r S rU=r$) ImageGPTPreTrainedModelifrM transformer input_ids)imageTrc z>[TU]U5 [U[5(aUR 5Hmup#SU;dM SU;dM[ R "USURR[R"SURR-5- S9 Mo g[U[5(aURRn[ R"UR[ R""[ R$"XD4[ R&S95R)SSXD55 gg) zInitialize the weights.r_r'grL)r4stdrIrN)r" _init_weightsrrnamed_parametersinitnormal_rMinitializer_rangemathr3n_layerrCrNcopy_rHr%rPrQrRrS)r(modulenameprer)s r*r%ImageGPTPreTrainedModel._init_weightsos f% fo . .!224t#D(8LL$++2O2ORVR[R[\]`d`k`k`s`s\sRt2tu5 1 2 2"MMAAM JJ  5::}&DEJJWX]]q- 3r,)r8r9r:r;r__annotations__base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_no_split_modulesr%no_gradrr?r@rAs@r*rrfsA %!O!&*#() ]]_r,rc~^\rSrSrS\4U4SjjrSrSr\SS\ RS-S\ S-S \ RS-S \ RS-S \ RS-S \ RS-S \ RS-S\ RS-S\ S-S\ S-S\ S-S\ S-S\ S\\-4Sjj5rSrU=r$) ImageGPTModelirMc Z>[TU]U5 URUl[R "UR UR5Ul[R "URUR5Ul [R"UR5Ul [R"[UR5Vs/sH n[!XS9PM sn5Ul[%URUR&S9UlSUlUR-5 gs snf)NrrF)r"r#rrTr Embedding vocab_sizewterNwper` embd_pdropdrop ModuleListrangenum_hidden_layersrhrrln_fgradient_checkpointing post_init)r(rMir)s r*r#ImageGPTModel.__init__s  ++<< 1 14>>B<< > >OJJv001 ERXRjRjLklLkq f BLklm%dnn&:S:ST &+#  msD(cUR$r!r)r(s r*get_input_embeddings"ImageGPTModel.get_input_embeddingss xxr,cXlgr!r )r(new_embeddingss r*set_input_embeddings"ImageGPTModel.set_input_embeddingss!r,Nrpast_key_valuesrytoken_type_ids position_ids inputs_embedsrrrroutput_hidden_states return_dictrr.c U bU OURRn U bU OURRn U bU OURRn U bU OURRn UbUb [ S5eUbFUR X5 UR5nURSUS5nURSnO1Ub#UR5SSnURSnO [ S5eUb URO URnUR(a/UR(aU (a[RS5 Sn UbURSUS5nU (aUc[URS9nUcAUbUR!5OSn["R$"USUS 9U-nUR'S5nUbURUS5nUcUR)U5nUR+U5nUUR-UR5-n[/URS S5(a[1URUUUS 9nO[3URUUS 9nUb[3URUUUS 9nUbUR)U5nUU-nUR5U5nUURS54-nU (aSOSnU (aURR6(aSOSnU (aSOSn[9UR:5H\unnU (aUU4-nU"UUUUUU U S9nUSnU (dM-UUS4-nURR6(dMSUUS4-nM^ UR=U5nUR"U6nU (aUU4-nU (d[?SUUUUU455$[AUUUUUS9$)a input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoImageProcessor`]. See [`ImageGPTImageProcessor.__call__`] for details. Examples: ```python >>> from transformers import AutoImageProcessor, ImageGPTModel >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> image_processor = AutoImageProcessor.from_pretrained("openai/imagegpt-small") >>> model = ImageGPTModel.from_pretrained("openai/imagegpt-small") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer0rz5You have to specify either input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)rM)rk is_decoder)rMrryr)rMrry)rMrryrr)rrrrrLc30# UH nUcMUv M g7fr!r).0vs r* (ImageGPTModel.forward..=swAws )last_hidden_staterr attentionscross_attentions)!rMrrrrrY%warn_if_padding_and_no_attention_maskrprSrrkrtraininglogger warning_oncer get_seq_lengthr%arange unsqueezerrtogetattrrr rr enumeraterrr<r)r(rrryrrrrrrrrrr input_shape batch_sizerkpast_seen_tokensposition_embedsrtoken_type_embeds output_shapeall_self_attentionsall_cross_attentionsall_hidden_statesrblockrs r*r6ImageGPTModel.forwards`2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKcd d  "  6 6y Q#..*K!r;r?;I"+J  &',,.s3K&,,Q/JTU U%.%:!! @T@T  & &4==##p"  %+00[_EN 0*$++>O  CRC^==?de  << BGJZZL'11!4L  %+00R@N   HHY/M((<0%(:(:=;O;O(PP 4;; e 4 4/{{+- / N7{{+-N " -%>{{+5&; & "  % $ 8 ),==M -0 "m&8&8&<%>> $5b4%64;;;Z;Zr`d"6BD!$&&)HAu#$58H$H!%'=#"3G$AJM  &9WQZM&I#;;222+?71:-+O(%*( -0 %**L9  1]4D D ':KM`bvw  9+++*1   r,)rrTrrrrr) NNNNNNNNNNNN)r8r9r:r;rr#r rrr%r&r rRrr<rr6r?r@rAs@r*rrsF~ "*.(,.2.2,0-1596:!%)-,0#'g <<$&g g  t+ g  t+ g llT) g ||d*g  %||d2g !& t 3g $;g  $;g #Tkg D[g g  : :g g r,rz The ImageGPT Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introc!^\rSrSrSS0rS\4U4Sjjr\SS\RS-S\ S-S \RS-S \RS-S \RS-S \RS-S \RS-S\RS-S\RS-S\ S-S\ S-S\ S-S\ S-S\ S\ \-4Sjj5rSrU=r$)ImageGPTForCausalImageModelingiLzlm_head.weightztransformer.wte.weightrMc>[TU]U5 [U5Ul[R "UR URS- SS9UlUR5 g)NrFrH) r"r#rrrLinearn_embdrlm_headrr(rMr)s r*r#'ImageGPTForCausalImageModeling.__init__UsL  (0yy0A0AA0EER  r,Nrrryrrrrrlabelsrrrrrr.c (U bU OURRn URUUUUUUUUU U U U S9 nUSnURU5nSnU brUSSS2SS24R 5nU SSS24R 5n[ 5nU"UR SURS55UR S55nU (dU4USS-nUbU4U-$U$[UUURURURURS9$)a input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoImageProcessor`]. See [`ImageGPTImageProcessor.__call__`] for details. labels (`torch.LongTensor` of shape `(batch_size, input_ids_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoImageProcessor, ImageGPTForCausalImageModeling >>> import torch >>> import matplotlib.pyplot as plt >>> import numpy as np >>> image_processor = AutoImageProcessor.from_pretrained("openai/imagegpt-small") >>> model = ImageGPTForCausalImageModeling.from_pretrained("openai/imagegpt-small") >>> device = torch.device("cuda" if torch.cuda.is_available() else "cpu") >>> model.to(device) # doctest: +IGNORE_RESULT >>> # unconditional generation of 8 images >>> batch_size = 4 >>> context = torch.full((batch_size, 1), model.config.vocab_size - 1) # initialize with SOS token >>> context = context.to(device) >>> output = model.generate( ... input_ids=context, max_length=model.config.n_positions + 1, temperature=1.0, do_sample=True, top_k=40 ... ) >>> clusters = image_processor.clusters >>> height = image_processor.size["height"] >>> width = image_processor.size["width"] >>> samples = output[:, 1:].detach().cpu().numpy() >>> samples_img = [ ... np.reshape(np.rint(127.5 * (clusters[s] + 1.0)), [height, width, 3]).astype(np.uint8) for s in samples ... ] # convert color cluster tokens back to pixels >>> f, axes = plt.subplots(1, batch_size, dpi=300) >>> for img, ax in zip(samples_img, axes): # doctest: +IGNORE_RESULT ... ax.axis("off") ... ax.imshow(img) ```N) rryrrrrrrrrrr.r0r)losslogitsrrrr ) rMrrr=rrrSrprrrrr )r(rrryrrrrrr@rrrrrtransformer_outputsr lm_logitsrB shift_logits shift_labelsloss_fctoutputs r*r6&ImageGPTForCausalImageModeling.forward]sLJ&1%L!#qr'?557L')HL--b,2C2CB2GH,J[J[\^J_`D\$7$;;F)-)9TGf$ Ev E0/??-;;*550AA   r,)r=r) NNNNNNNNNNNNN)r8r9r:r;_tied_weights_keysrr#rr%r&r rRrr<rr6r?r@rAs@r*r8r8Lsa+,DE~*.(,.2.2,0-1596:&*!%)-,0#'l <<$&l l  t+ l  t+ l llT) l ||d*l  %||d2l !& t 3l  t#l $;l  $;l #Tkl D[l l 2 2!l l r,r8z The ImageGPT Model transformer with an image classification head on top (linear layer). [`ImageGPTForImageClassification`] average-pools the hidden states in order to do the classification. cR^\rSrSrS\4U4Sjjr\SS\RS-S\ S-S\RS-S\RS-S \RS-S \RS-S \RS-S \ S-S \ S-S\ S-S\ S-S\ S\ \ -4Sjj5rSrU=r$)ImageGPTForImageClassificationirMc>[TU]U5 URUl[U5Ul[ R "URURSS9UlUR5 g)NFr:) r"r# num_labelsrrrr;r<scorerr>s r*r#'ImageGPTForImageClassification.__init__sR   ++(0YYv}}dooEJ  r,Nrrryrrrr@rrrrrr.c U bU OURRn URUUUUUUUU U U S9 n U SnURSS9nUR U5nSnUbUR UUUR5nU (dU4U SS-nUbU4U-$U$[ UUU RU RU RS9$)a input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoImageProcessor`]. See [`ImageGPTImageProcessor.__call__`] for details. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, ImageGPTForImageClassification >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> image_processor = AutoImageProcessor.from_pretrained("openai/imagegpt-small") >>> model = ImageGPTForImageClassification.from_pretrained("openai/imagegpt-small") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits ```N) rryrrrrrrrrrrl)rBrCrrr) rMrrr4rP loss_functionrrrr)r(rrryrrrr@rrrrrrDrpooled_hidden_statesrCrBrIs r*r6&ImageGPTForImageClassification.forwardsf&1%rhs@%  %&!CC)J9 .# ,2   H %  M) M)`"))"E*.E*PoD +  D x %