3jLSrSSKJr SSKrSSKJr SSKJr SSKJr SSK J r SS K J r J r Jr SS KJr SS KJrJr SS KJr SS KJr SSKJrJrJrJrJrJr SSKJ r SSK!J"r" \RF"\$5r%\"SS9\"SS\555r&\"SS9\"SS\555r'"SS\RP5r)\"SS\55r*\"SS9"SS \*55r+\"SS9"S!S"\*\ 55r,/S#Qr-g)$zPyTorch PaliGemmamodel.) dataclassN)nn)Cache)PreTrainedConfig)GenerationMixin)create_causal_maskcreate_masks_for_generate!create_sliding_window_causal_mask)FlashAttentionKwargs)BaseModelOutputWithPastBaseModelOutputWithPooling)PreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tupleloggingtorch_compilable_check) AutoModel)PaliGemmaConfigzN Base class for Paligemma outputs, with hidden states and attentions.  custom_introcB\rSrSr%SrSr\RS-\S'Sr g)PaligemmaModelOutputWithPast,a image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states) __name__ __module__ __qualname____firstlineno____doc__r torch FloatTensor__annotations____static_attributes__r!j/home/wildlama/miniconda3/lib/python3.13/site-packages/transformers/models/paligemma/modeling_paligemma.pyrr,s 59**T18r+rzU Base class for PaliGemma causal language model (or autoregressive) outputs. c\rSrSr%SrSr\RS-\S'Sr \RS-\S'Sr \ S-\S'Sr \ \RS-\S'Sr\ \RS-\S'Sr\RS-\S 'S rg) PaliGemmaCausalLMOutputWithPast<a loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.text_config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder after projecting last hidden state. Nlosslogitspast_key_values hidden_states attentionsr r!)r"r#r$r%r&r0r'r(r)r1r2rr3tupler4r r*r!r+r,r.r.<s &*D%  d ")'+FE   $+$(OUT\(59M5**+d2926Je''(4/648**T18r+r.c6^\rSrSrS\4U4SjjrSrSrU=r$)PaliGemmaMultiModalProjectorZconfigc>[TU]5 [R"URR URR SS9Ulg)NTbias)super__init__rLinear vision_config hidden_sizeprojection_dimlinearselfr9 __class__s r,r>%PaliGemmaMultiModalProjector.__init__[s; ii 4 4 @ @&BVBVBeBelpq r+c(URU5nU$NrC)rEimage_featuresr3s r,forward$PaliGemmaMultiModalProjector.forward_s N3 r+rJ) r"r#r$r%rr>rLr* __classcell__rFs@r,r7r7Zsrrr+r7cL\rSrSr%\\S'SrSrSrS/r S/r Sr Sr Sr SrSrS rg ) PaliGemmaPreTrainedModeler9model)imagetextTr7r2Fr!N)r"r#r$r%rr)base_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_can_compile_fullgraph_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendr*r!r+r,rQrQesI (&*#78#4"5"N"&r+rQz| The Base Paligemma model which consists of a vision backbone and a language model without language modeling head., c^\rSrSrSrS\4U4Sjjr\\"SS9S\ RS\ \ S \ \-4S j55rS \ R S \ RS \ R4Sjr\\SS \ R S-S\ RS-S\ R$S-S\ R S-S\S-S\ R S-S \ RS-S\ R S-S\S-S\ \S \ \-4Sjj55rSrU=r$)PaliGemmaModeltFr9c>[TU]U5 [R"URS9Ul[ U5UlURRUl [R"URS9nX l URR5R=(d URUlUR5 g)N)r9)r=r>r from_configr@ vision_towerr7multi_modal_projector text_config vocab_sizelanguage_modelr9get_text_configdtypetext_config_dtype post_init)rEr9rirFs r,r>PaliGemmaModel.__init__}s  %119M9MN%A&%I" ,,77"..f6H6HI,!%!!D!D!R  r+zWObtains image last hidden states from the vision tower and apply multimodal projection.r pixel_valueskwargsreturnc rUR"U40UD6nURnURU5nXSlU$rI)relast_hidden_staterf pooler_output)rErorp image_outputsselected_image_featurerKs r,get_image_features!PaliGemmaModel.get_image_featuressB)),A&A !.!@!@334JK&4#r+ input_ids inputs_embedsrKc &UcjX R5"[R"URR[R UR S95:HnURS5nOXRR:HnUR5nURSURS-nURS5RUR 5n[XRRS-UR5:HSUSU35 U$)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. )rkdevicerrz6Image features and image tokens do not match, tokens: z , features: )get_input_embeddingsr'tensorr9image_token_idlongr|allsumshape unsqueezetornumel)rEryrzrKspecial_image_maskn_image_tokensn_image_featuress r,get_placeholder_mask#PaliGemmaModel.get_placeholder_masks  !.2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*kk.H.H!H +//1)//2^5I5I!5LL/99"=@@AUAUV 004 48L8L8N ND^DTT`aq`r s "!r+Nattention_mask position_idsr2token_type_idslabels use_cachec USLUSL- (a [S5eUbQURRUR:a-XRR:Hn UR 5n SX'OUn UcUR 5"U 5nUcXUbUR 5OSn [R"URSURS9U -nURS5S-nUbbURU5RnURURUR5nUR!XUS9n UR#X5nURR%5UUUUS.nUSL=(d UR&(+=(d USLnUb%U(a[R("US:HSS5US '[+S0UD6n[-URR.S S5bUR15nU[3S0UD6S .nUR4"SUUUUU S .U D6n[7UR8UR:UR<UR>UbWS 9$SS 9$)  labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.text_config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.text_config.vocab_size]`. Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, PaliGemmaForConditionalGeneration >>> model = PaliGemmaForConditionalGeneration.from_pretrained("google/paligemma2-3b-mix-224") >>> processor = AutoProcessor.from_pretrained("google/paligemma2-3b-mix-224") >>> prompt = "Where is the cat standing?" >>> url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, text=prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(**inputs,) >>> processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Where is the cat standing?\nsnow" ```Nz:You must specify exactly one of input_ids or inputs_embedsrrr|)rzrK)r9rzrr2rr}block_sequence_idssliding_window)full_attentionsliding_attention)rrr2rzr)rsr2r3r4r r!) ValueErrorr9rrhcloner~get_seq_lengthr'arangerr|rrwrtrrkrmasked_scatterrjis_initializedwherer getattrrgcopyr rirrsr2r3r4)rEryrorrr2rrzrrrpr llm_input_idspast_seen_tokensrK mask_kwargsis_first_iteration causal_masksliding_mask_kwargsoutputss r,rLPaliGemmaModel.forwardsZ  -t"; <YZ Z  T[[%?%?4??%R!*kk.H.H!H %OO-M01M -%M   557 FM  CRC^==?de  << (;(;A(>}G[G[\_ooL'11!4q8L  #!44\BPPN+..}/C/C]EXEXYN!%!:!:~";" *889K\Mkk113*,.(  -4vO2J   QU   r+)rirfrlrerh) NNNNNNNNN)r"r#r$r%accepts_loss_kwargsrr>rrr'r(rrr5rrw LongTensorrTensorrboolr rrLr*rNrOs@r,rarats   n!--9?@R9S + +"))":?:K:K"]b]n]n"0.215.204(,2626*.!%l ##d*l ''$.l  t+ l &&- l  l ((4/l ((4/l   4'l $;l -.l  - -l l r+rac^\rSrSrSS0rS\4U4Sjjr\S\RS\ \ 4Sj5r \ \SS \RS -S\RS -S \RS -S \RS -S \S -S\RS -S\RS -S\RS -S\S -S\\R-S\ \ S\\-4Sjj55rSU4Sjjr\SS\S\RS \RS -S \S -S \RS -S\RS -S\S -S\4Sjj5rSrU=r$)!PaliGemmaForConditionalGenerationi zlm_head.weightz(model.language_model.embed_tokens.weightr9c>[TU]U5 [U5Ul[R "UR RUR RSS9Ul UR5 g)NFr;) r=r>rarSrr?rgrArhlm_headrmrDs r,r>*PaliGemmaForConditionalGeneration.__init__(sS  #F+ yy!3!3!?!?ASASA^A^ejk  r+rorpc <URR"U40UD6$rI)rSrw)rErorps r,rw4PaliGemmaForConditionalGeneration.get_image_features.szz,,\DVDDr+Nryrrr2rrzrrlogits_to_keeprqc UR"SUUUUUUUU US. U D6n U Sn [U [5(a [U *S5OU nUR U SS2USS245nSnUb3UR "SXUR RRS.U D6n[UUU RU RU RU RS9$)r) ryrorrrr2rzrrrN)r1rrh)r0r1r2r3r4r r!)rS isinstanceintslicer loss_functionr9rgrhr.r2r3r4r )rEryrorrr2rrzrrrrprr3 slice_indicesr1r0s r,rL)PaliGemmaForConditionalGeneration.forward2sZ**  %))%+'      8B>SV8W8W~ot4]k mA}a,?@A  %%9P9P9[9[_eD/#33!//)) ' ; ;   r+c >[TU]"U4UUUUUU UU S.U D6n U RS5b U SS-U S'U (dU(dX]S'U $)N)r2rzrrrrrrrrro)r=prepare_inputs_for_generationget)rEryr2rzrrorrrrrrrp model_inputsrFs r,r?PaliGemmaForConditionalGeneration.prepare_inputs_for_generationsz w<   +')%))1       N + 7+7+G!+KL ( Y+7 (r+rc [R"/UR5SSQSURS9nUb"US:HSS5n[ UR 5UUUUUS9$)Nr}rr)r9rzrrr2r)r'fullsizer|rr rj) r9rzrr2rrrrp group_idss r,r ;PaliGemmaForConditionalGeneration.create_masks_for_generatesyJJ;!3!3!5cr!:;R H\H\]  % Na$7B?I())+'()+%   r+)rrS) NNNNNNNNNr) NNNNNNTNNF)NF)r"r#r$r%_tied_weights_keysrr>rr'r(rrrwrrrrrrr5r.rLr staticmethodrdictr r*rNrOs@r,rr s- +,VW Eu/@/@EFSeLfEE.215.204(,2626*.!%-.J ##d*J ''$.J  t+ J &&- J  J ((4/J ((4/J   4'J $;J ell*J +,J  0 0J J ^ )V/3*/   ||  t+   llT)   t+  !4K    r+r)rrQra).r& dataclassesrr'r cache_utilsrconfiguration_utilsr generationr masking_utilsr r r modeling_flash_attention_utilsr modeling_outputsr rmodeling_utilsrprocessing_utilsrutilsrrrrrrautorconfiguration_paligemmar get_loggerr"loggerrr.Moduler7rQrar__all__r!r+r,rs:!  3)mmBS-&4   H %  9#:9  9  9k9  90299 ' ' ' d -d  d N ^ (@/^  ^ B ^r+