3jISrSSKrSSKJr SSKJr SSKJr SSKrSSKJ r SSK J r SS K Jr SS KJr SS KJr SS KJr SS KJr SSKJrJrJrJrJrJr SSKJrJ r SSK!J"r" SSK#J$r$ SSK%J&r&J'r'J(r(J)r)J*r*J+r+ SSK,J-r- SSK.J/r/J0r0 SSK1J2r2J3r3J4r4 SSK5J6r6J7r7J8r8 \*Rr"\:5r;\(\"SS\555r<\("SS9\"SS\&555r="SS \ R|5r?SNS!\ R|S"\RS#\RS$\RS%\RS-S&\AS'\A4S(jjrB"S)S*\ R|5rC"S+S,\ R|5rD"S-S.\5rE\("S/S0\ 55rF"S1S2\ R|5rG"S3S4\F5rH"S5S6\ R|5rI"S7S8\ R|5rJ"S9S:\ R|5rK"S;S<\ R|5rL"S=S>\ R|5rM"S?S@\5rN"SASB\ R|5rO"SCSD\ R|5rP"SESF\F5rQ\("SGS9"SHSI\F55rR\("SJS9"SKSL\F\55rS/SMQrTg)OzPyTorch InstructBLIP model.N)Callable) dataclass)Any)nn)initialization)ACT2FN)GenerationMixin)create_bidirectional_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsBaseModelOutputWithPooling,BaseModelOutputWithPoolingAndCrossAttentionsCausalLMOutputWithPastSeq2SeqLMOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging torch_int)merge_with_config_defaults)OutputRecordercapture_outputs) AutoModelAutoModelForCausalLMAutoModelForSeq2SeqLM)InstructBlipConfigInstructBlipQFormerConfigInstructBlipVisionConfigcB\rSrSr%SrSr\S-\S'Sr\ S-\S'Sr g)'BaseModelOutputWithVisionQformerOutputs3z vision_outputs (`BaseModelOutputWithPooling`): Outputs of the vision encoder. qformer_outputs (`BaseModelOutputWithPoolingAndCrossAttentions`): Outputs of the Q-Former (Querying Transformer). Nvision_outputsqformer_outputs) __name__ __module__ __qualname____firstlineno____doc__r,r__annotations__r-r__static_attributes__r.p/home/wildlama/miniconda3/lib/python3.13/site-packages/transformers/models/instructblip/modeling_instructblip.pyr*r*3s)9=N.5# UH*nUS;aTUO[TU5R5v M, g7f)r,r-r>N)getattrto_tuple).0kselfs r7 KInstructBlipForConditionalGenerationModelOutput.to_tuple..]sC !WW Gq!**, -!s25)tuplekeysrGs`r7rD8InstructBlipForConditionalGenerationModelOutput.to_tuple\s% YY[    r6r.)r/r0r1r2r3r<rJtorch FloatTensorr4r=r,rr-rr>rrrrDr5r.r6r7r:r:As -1D%!! "T )0.2FE%## $t +28[TU]5 XlURUlUR UlUR Ul[R"[R"SSUR55Ul [R"SURUR UR S9Ul UR UR -S-UlURS-Ul[R"[R"SURUR55Ulg)Nr%r) in_channels out_channels kernel_sizestrider!)super__init__rS hidden_size embed_dim image_size patch_sizer ParameterrNrandnclass_embeddingConv2dpatch_embedding num_patches num_positionsposition_embeddingrGrS __class__s r7rZ%InstructBlipVisionEmbeddings.__init__gs  ++ ++ ++!||EKK1dnn,MN!yyDOO\`\k\k !OOt>1D!--1"$,,u{{1d>P>PRVR`R`/a"br6 embeddingsheightwidthr?cURSS- nURRSS- n[RR 5(dXE:XaX#:Xa UR$URSS2SS24nURSS2SS24nURSnX R -n X0R -n [ US-5n URSXU5nURSSSS5n[RRUX4SS S 9nURSSSS5RSSU5n[R"Xg4SS 9$) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 r%Ng?rrr!bicubicF)sizemode align_cornersdim)shaperfrNjit is_tracingr^rreshapepermuter functional interpolateviewcat) rGrjrkrlrdreclass_pos_embedpatch_pos_embedrt new_height new_widthsqrt_num_positionss r7interpolate_pos_encoding5InstructBlipVisionEmbeddings.interpolate_pos_encodingysS!&&q)A- //55a81< yy##%%+*F6?** *11!RaR%811!QR%8r". __, &}c'9:)11!5G]`a)11!Q1=--33 ( 4 *11!Q1=BB1b#Nyy/;CCr6 pixel_valuesrcURup4pVURRRnURUR US95nUR S5R SS5nURRUSS5R U5n [R"X/SS9n U(aURXU5n O URn XSS2SU RS52SS24R U5-n U $)N)dtyper!r%rnrs)rurcweightrtoflatten transposeraexpandrNr}rrfrp) rGrr batch_size_rkrl target_dtype patch_embeds class_embedsrjrfs r7forward$InstructBlipVisionEmbeddings.forwards'3'9'9$ v++2288 ++LOO,O,OP #++A.88A> ++22:q"EHHV YY ;C #!%!>!>zSX!Y !%!8!8 Q8L*//!:L8La5O"P"S"ST`"aa r6) rarSr\r]rdrercr^rfF)r/r0r1r2r(rZrNTensorintrrOboolrr5 __classcell__rhs@r7rQrQfsrc7c$&D5<<&D&DUX&D]b]i]i&DP E$5$5 QU bgbnbn  r6rQmodulequerykeyvalueattention_maskscalingdropoutc `[R"XRSS55U-nUbX-n[RR USS9n[RR XURS9n[R"X5n U RSS5R5n X4$)Nrnrs)ptrainingr%r!) rNmatmulrrrzsoftmaxrr contiguous) rrrrrrrkwargs attn_weights attn_outputs r7eager_attention_forwardrs<<}}R'<=GL!#4 ==((2(>L==((6??([L,,|3K''1-88:K  $$r6c^\rSrSrSrU4SjrS\RS\S\4Sjr S\RS \ \R\RS -\ \RS -44S jr S r U=r $) InstructBlipAttentionz=Multi-headed attention from 'Attention Is All You Need' paperc>[TU]5 XlURUlUR UlURUR -UlURUR -UR:wa&[SURSUR S35eURS-Ul SUl URUl [R"URSUR-SS9UlUR(ai[R "["R$"UR55n[R "["R$"UR55nOSnSnUbQ["R&"U["R("USS9U45n[R "U5URl[R"URUR5Ulg) Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gFr)bias) requires_grad)rYrZrSr[r\num_attention_heads num_headshead_dim ValueErrorscale is_causalattention_dropoutrLinearqkvqkv_biasr_rNzerosr} zeros_liker projection)rGrSq_biasv_biasrrhs r7rZInstructBlipAttention.__init__ss  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( !'!9!999T^^Q-?eL ??\\%++dnn"=>F\\%++dnn"=>FFF  yy&%*:*:6QV*WY_!`aHLL2DHHM))DNNDNNCr6tensorseq_lenbszcURX2URUR5RSS5R 5$)Nr%r!)r|rrrr)rGrrrs r7_shapeInstructBlipAttention._shapes5{{3GQQRSUVWbbddr6 hidden_statesr?Nc UR5up4nURU5nURX4SURXPR-5R SSSSS5nUSUSUSpn[ R "URR[5n U "UUUU 4SUR(dSO URURS.UD6upU RX4S 5R5n URU 5n X4$) z#Input shape: Batch x Time x Channelrr!rr%N)rrrrn)rprrxrryr get_interfacerS_attn_implementationrrrrrr) rGrrrtgt_lenr\ mixed_qkv query_states key_states value_statesattention_interfacerrs r7rInstructBlipAttention.forwards#0"4"4"6iHH]+ %%cAt~~yTbTbGbckk q!Q 2;1y|YWX\, (?(M(M KK , ,.E) %8     %  #}}C$2H2HJJ %  % ! "))#;FFH ook2 ((r6) rrSr\rrrrrr)r/r0r1r2r3rZrNrrrrJrr5rrs@r7rrsuGD>eU\\eCece")||") u||U\\D0% 2E2LL M ")")r6rcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)InstructBlipMLPic>[TU]5 Xl[URUl[ R"URUR5Ul [ R"URUR5Ul gN) rYrZrSr hidden_act activation_fnrrr[intermediate_sizefc1fc2rgs r7rZInstructBlipMLP.__init__sb  #F$5$5699V//1I1IJ99V55v7I7IJr6rr?clURU5nURU5nURU5nU$r)rrrrGrs r7rInstructBlipMLP.forwards4/ **=9 / r6)rrSrr r/r0r1r2rZrNrrr5rrs@r7rrs)KU\\ellr6rc~^\rSrSrS\4U4Sjjr\S\RS\ \ S\R4Sj5r Sr U=r$) InstructBlipEncoderLayeri$rSc<>[TU]5 URUl[ U5Ul[ R"URURS9Ul [U5Ul [ R"URURS9Ul gNeps) rYrZr[r\r self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2rgs r7rZ!InstructBlipEncoderLayer.__init__%sm ++.v6<<F> '   &0  ((7 / %0 r6)r\rrrr)r/r0r1r2r&rZrrNrrrrOrr5rrs@r7rr$sRS1S||+,    r6rc^\rSrSr%\\S'SrSrSrSr Sr Sr Sr Sr /SQr\R "5U4Sj5rSrU=r$) InstructBlipPreTrainedModeliDrSblip)imagetextT)InstructBlipQFormerEmbeddingsrrInstructBlipQFormerLayer%InstructBlipQFormerMultiHeadAttentionInstructBlipQFormerSelfOutputcV>[TU]U5 URRn[ U[ 5(aA[ R"URSUS9 [ R"URSUS9 g[ U[[45(a![ R"UR5 g[ U[5(a\[ R"UR ["R$"UR R&S5R)S55 gg)zInitialize the weightsr)meanstdrnr%rnN)rY _init_weightsrSinitializer_range isinstancerQinit trunc_normal_rfra$InstructBlipForConditionalGenerationInstructBlipModelzeros_ query_tokensrcopy_ position_idsrNarangerur)rGrfactorrhs r7r)InstructBlipPreTrainedModel._init_weightsZs f%.. f: ; ;   v88s O   v55CV L !EGX Y Z Z KK++ ,  = > > JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h?r6r.)r/r0r1r2r&r4base_model_prefixinput_modalitiessupports_gradient_checkpointing_supports_attention_backend_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_no_split_modulesrNno_gradrr5rrs@r7rrDs\ (&*#"&N! ]]_ i ir6rc\^\rSrSrSrS\4U4Sjjr\S\\ S\ \ -4Sj5r Sr U=r$) InstructBlipEncoderiiz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`InstructBlipEncoderLayer`]. Args: config (`InstructBlipConfig`): The corresponding vision configuration for the `InstructBlipEncoder`. rSc>[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5UlSUl gs snfNF) rYrZrSr ModuleListrangenum_hidden_layersrlayersgradient_checkpointing)rGrSrrhs r7rZInstructBlipEncoder.__init__ssT  mmuU[UmUmOn$oOn!%=f%EOn$op &+#%pA&rr?c PUnURH nU"U40UD6nM [US9$)Nlast_hidden_state)rr)rG inputs_embedsrr encoder_layers r7rInstructBlipEncoder.forwardys9 & ![[M)M) ??r6)rSr r)r/r0r1r2r3r&rZrrrrJrrr5rrs@r7rrisL,1,  @+, @  @ @r6rc ^\rSrSr%SrSr\\S'\\ S.r S\4U4Sjjr \ \ "SS9\SS\R S -S \S \\S \\-4S jj555rSrSrU=r$)InstructBlipVisionModelir)rrS)r attentionsc>[TU]U5 XlURn[ U5Ul[ U5Ul[R"X!RS9Ul UR5 gr) rYrZrSr[rQrjrencoderrrrpost_layernorm post_init)rGrSr\rhs r7rZ InstructBlipVisionModel.__init__sY   && 6v>*62  ll9:O:OP r6F)tie_last_hidden_statesNrrr?c Uc [S5eURXS9nUR"SSU0UD6nURnUR U5nUSS2SSS24nUR U5n[ UUS9$)Nz You have to specify pixel_values)rr&rr% pooler_outputr.)rrjr-r%r.r)rGrrrrencoder_outputsr% pooled_outputs r7rInstructBlipVisionModel.forwards  ?@ @ h +/<<, ', ,  ,== //0AB)!Q'2 ++M: )/'  r6cUR$r)rjrLs r7get_input_embeddings,InstructBlipVisionModel.get_input_embeddingss r6)rSrjr-r.r)r/r0r1r2main_input_namerr(r4rr_can_record_outputsrZrr rrNrOrrrrJrrr9r5rrs@r7r*r*s$O! $$1+ 7  E226). ''$. #' +,  + +  3  6r6r*ch^\rSrSrS U4SjjrSrSrSrSrSr S S\ \ 4S jjr S r U=r$) ric>[TU]5 XlURUR-S:wa5[ US5(d$[ SURUR4-5eURUl[URUR- 5UlURUR-Ul [R"URUR5Ul U(aa[R"URUR5Ul[R"URUR5UlO`[R"URUR5Ul[R"URUR5Ul[R "UR"5UlSUlg)Nrembedding_sizezLThe hidden size (%d) is not a multiple of the number of attention heads (%d)F)rYrZrSr[rhasattrrrattention_head_size all_head_sizerrrencoder_hidden_sizerrDropoutattention_probs_dropout_probrsave_attentionrGrSis_cross_attentionrhs r7rZ.InstructBlipQFormerMultiHeadAttention.__init__sb     : : :a ?PVXhHiHi^%%v'A'ABC  $*#=#= #&v'9'9F 11$((;P2QRI33DJJ?T4UVK3N11$((=2IJI33DJJ}4MNK JJ}5//0AB !<< 5H5HR5PQ+dii8P8P.QQ!1!7!7  %/@ **,-=>AABXY $"5"5  # #O 4  ) )$*B*B C#',,"? %<J %--aAq9DDF "/"4"4"6s";t?Q?Q>S"S%**,CD --r6) rBrArTrLrSrrrrrFrrNNN)r/r0r1r2rZrMrPrUrXr]rrrr5rrs@r7rrsF$0-#+"%"# 4. +, 4.4.r6rcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $)ri!c(>[TU]5 [R"URUR5Ul[R "URURS9Ul[R"UR5Ul gr) rYrZrrr[denserrrDhidden_dropout_probrrgs r7rZ&InstructBlipQFormerSelfOutput.__init__"s` YYv1163E3EF f&8&8f>S>STzz&"<"<= r6r input_tensorr?cpURU5nURU5nURX-5nU$rrtrrrGrrws r7r%InstructBlipQFormerSelfOutput.forward(5 =1  ]3 }'CD r6rrtrrrs@r7rr!6> U\\RWR^R^r6rc ^\rSrSrS U4SjjrS S\R S\RS-S\RS-S\RS-S\\ S \R 4 S jjr S r U=r $)InstructBlipQFormerAttentioni0cb>[TU]5 [X5Ul[ U5Ulgr)rYrZr attentionroutputrGs r7rZ%InstructBlipQFormerAttention.__init__1s& >vZ3F; r6Nrrrdrerr?c ZUR"SUUUUS.UD6upgURXa5nU$)N)rrrdrer.rr) rGrrrdrerrrattention_outputs r7r$InstructBlipQFormerAttention.forward6sF ')"7#9      ;;{Br6rrrq) r/r0r1r2rZrNrrOrrrr5rrs@r7rr0s<48:>;?  || ))D0  %0047  !& 1 1D 8  +,     r6rcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)InstructBlipQFormerIntermediateiJc>[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl gr) rYrZrrr[rrtrrstrr intermediate_act_fnrgs r7rZ(InstructBlipQFormerIntermediate.__init__Ks` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $r6rr?cJURU5nURU5nU$rrtrrs r7r'InstructBlipQFormerIntermediate.forwardSs& =1 00? r6rrrs@r7rrJs(9U\\ellr6rcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $)InstructBlipQFormerOutputiZc(>[TU]5 [R"URUR 5Ul[R"UR URS9Ul[R"UR5Ul gr) rYrZrrrr[rtrrrDrurrgs r7rZ"InstructBlipQFormerOutput.__init__[s` YYv779K9KL f&8&8f>S>STzz&"<"<= r6rrwr?cpURU5nURU5nURX-5nU$rryrzs r7r!InstructBlipQFormerOutput.forwardar|r6r}rrs@r7rrZr~r6rcT^\rSrSrU4SjrSS\\4SjjrSrSr Sr U=r $) rihc^>[TU]5 URUlSUl[ U5UlX lX!R-S:Xa[ USS9UlSUl OSUl [U5Ul [U5Ul [U5Ul[U5Ulg)Nr%rT)rHF)rYrZchunk_size_feed_forward seq_len_dimrr layer_idxcross_attention_frequencycrossattentionhas_cross_attentionr intermediaterrintermediate_query output_queryrGrSrrhs r7rZ!InstructBlipQFormerLayer.__init__is '-'E'E$5f=" 77 71 <">vZ^"_D '+D $',D $;FC/7 "A&"I5f=r6rc lUR"U4SU0UD6nUS:aUSS2SU2SS24nUR(a%Uc [S5eUR"U4UUUS.UD6n[ UR UR URU5n URSU:ag[ URUR URUSS2US2SS245RU R5n [R"X/SS9n U $[ URUR URU5n U $)Nrrz>encoder_hidden_states must be given for cross-attention layers)rrdrer%rs)rrrrrfeed_forward_chunk_queryrrrufeed_forward_chunkrdevicerNr}) rGrrrdre query_lengthrrquery_attention_output layer_outputlayer_output_texts r7r InstructBlipQFormerLayer.forward}si >>  )   ! %5a,6I%J "''(0$%eff)-)<)<**#1*?+A *  *&5--,,  & L %%a(<7$=++00$$$Q q%89 % "\(() " %yy,)JPQR  5'',,   L r6cJURU5nURX!5nU$r)rrrGrintermediate_outputrs r7r+InstructBlipQFormerLayer.feed_forward_chunks)"//0@A{{#6I r6cJURU5nURX!5nU$r)rrrs r7r1InstructBlipQFormerLayer.feed_forward_chunk_querys+"556FG(()<O r6) rrrrrrrrrrNNNr) r/r0r1r2rZrrrrrr5rrs@r7rrhs;>."# 3+,3j r6rcR^\rSrSrU4Sjr\SS\\4Sjj5rSr U=r $)InstructBlipQFormerEncoderic >[TU]5 Xl[R"[ UR 5Vs/sHn[X5PM sn5UlSUl gs snfr) rYrZrSrrrrrlayerr rs r7rZ#InstructBlipQFormerEncoder.__init__sY  ]]JOPVPhPhJi jJiY %f 8Ji j  ',# kr"rc [URR5H nURUnU"UUU4UUS.UD6nM" [ US9$)N)rerr$)rrSrrr) rGrrrdrerri layer_modules r7r"InstructBlipQFormerEncoder.forwardsft{{445A::a=L(%(>)   M69+  r6)rSr rr) r/r0r1r2rZrrrrr5rrs@r7rrs:,"#  +,  r6rc>^\rSrSrSrU4SjrSSjrSrU=r$)riz;Construct the embeddings from word and position embeddings.c">[TU]5 [R"URUR UR S9Ul[R"URUR 5Ul [R"UR URS9Ul [R"UR5UlUR!S["R$"UR5R'S5SS9 Xlg)N) padding_idxrr rF) persistent)rYrZr Embedding vocab_sizer[ pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingsrr layernormrDrurregister_bufferrNr rrSrgs r7rZ&InstructBlipQFormerEmbeddings.__init__s !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej   r6cUbUR5SnOSnUc%URSS2XEU-24R5nUbYURU5nUR UR UR 55nXg-nUb[R"X64SS9nOUnUR URRR5nURU5nURU5nU$)Nr%rrs) rpr clonerrrrrNr}rrrr)rG input_idsr  query_embedspast_key_values_length seq_lengthrjrs r7r%InstructBlipQFormerEmbeddings.forwards  ")!,JJ  ,,Q0FVlIl0l-lmssuL  --i8J"&":":>#8#8#>#>? ^^J/ \\*- r6)rSrrrrr) r/r0r1r2r3rZrr5rrs@r7rrs#E " r6rc^\rSrSrSrSrSrSrSr\ \ "\ SSS9/\ "\ SSS9/S.r S \ 4U4S jjrS rS r\\\SS\R*S\R,S -S\R*S -S\R.S -S\R,S -S\R,S -S\\S\\R,\-4Sjj555rSrU=r$)InstructBlipQFormerModeliz Querying Transformer (Q-Former), used in InstructBLIP. Slightly modified from BLIP-2 as it also takes the instruction as input. Fr%z .attention)index layer_namez.crossattention)rr+cross_attentionsrSc>[TU]U5 Xl[U5Ul[ U5UlUR5 gr)rYrZrSrrjrr-r/rgs r7rZ!InstructBlipQFormerModel.__init__)s7   7?1&9  r6c.URR$rrjrrLs r7r9-InstructBlipQFormerModel.get_input_embeddings3s...r6c$XRlgrr)rGrs r7set_input_embeddings-InstructBlipQFormerModel.set_input_embeddings6s */'r6Nrrr rrdrerr?c >UcUc [S5eUbURSOSnURUUUS9n [URU US9nUb[URU UUS9nUR "U 4UUUUS.UD6n U R n U SS2SSS24n [U U S 9$) a query_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Hidden states to be used in the attention computation. If cross-attention, will be used for the query (i.e., key and value will use the encoder_hidden_states). Nz7You have to specify query_embeds when input_ids is Noner%r)rr r)rSr&r)rSr&rrd)rrdrerr3)rrurjr rSr-r%r) rGrrr rrdrerrembedding_outputr5sequence_outputr6s r7r InstructBlipQFormerModel.forward9s$  !5VW W0<0H|))!,a ??%%+  3;;*)  " -%>{{.5&; & ",0<< , )"7#9% ,  , *;;'1a0 ;-'  r6)rSrjr-)NNNNN)r/r0r1r2r3rrrrrrrr<r'rZr9rrr rrN LongTensorrOrrrrJrrr5rrs@r7rrsB #( N2 @Vb c  @Vg h 8/0 4804,0:>;?6 ##6 ))D06 &&- 6 llT) 6 %0047 6 !& 1 1D 86 +,6  u  !$P P6  6 r6rz[ InstructBLIP base Model consisting of language model, qformer and vision encoder. c^\rSrSrSrS/rS\4U4SjjrSrS\ RS\ R4S jr \ \SS\ RS \ RS \ RS -S\ RS -S \ RS -S\ RS -S\ RS -S\ RS -S\S\\S\\-4Sjj55rSrU=r$)riurrrSc>[TU]U5 [UR5Ul[ R "[R"SURURR55Ul [UR5Ul[ R"URRUR R5Ul[$R&"UR 5UlUR+5 gNr%)rYrZr* vision_config vision_modelrr_rNrnum_query_tokensqformer_configr[rrqformerr text_configlanguage_projectionr" from_configlanguage_modelr/rgs r7rZInstructBlipModel.__init__~s  3F4H4HILLQ8O8OQWQfQfQrQr)st/0E0EF #%99V-B-B-N-NPVPbPbPnPn#o '33F4F4FG r6c"URn[U5S:a=SU;a7[RR 5S:a[ R S5 [URS5(aSURRl ggz Some pre-processing hacks to make the model `accelerate` compatible. Check https://github.com/huggingface/transformers/pull/21707 for more details. r%raThe `language_model` is not in the `hf_device_map` dictionary and you are running your script in a multi-GPU environment. this may lead to unexpected behavior when using `accelerate`. Please pass a `device_map` that contains `language_model` to remove this warning. Please refer to https://github.com/huggingface/blog/blob/main/accelerate-large-models.md for more details on creating a `device_map` for large models._hf_hookTN hf_device_maplenrNcuda device_countloggerwarningr@rrio_same_devicerGrs r7_preprocess_accelerate(InstructBlipModel._preprocess_accelerate| ** }  !&6m&KPUPZPZPgPgPilmPm NNM  4&& 3 3:>D   ( ( 7 4r6rr&c dUcjX R5"[R"URR[R UR S95:HnURS5nOXRR:HnURS5RUR 5nU$zJ Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`. rrrn r9rNrrSimage_token_idlongrall unsqueezerrGrr&special_image_masks r7get_placeholder_mask&InstructBlipModel.get_placeholder_mask  !.2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*kk.H.H!H /99"=@@AUAUV!!r6Nqformer_input_idsqformer_attention_maskrdecoder_input_idsdecoder_attention_maskrrr?c XUR"S UU S.U D6n U Sn [R"U R5SS[RU R S9n UR RU RSSS5n[R"UR5SS[RU R S9nUc[R"U5nURUR 5n[R"X/SS9nUR"S UUUU U S.U D6nUSSS2SURS52SS24nUc9URR5"U5nUc[R"U5nURU5nURUR UR 5nUR#XHS 9nUR%UU5nUR&R((aUR"S UUS .U D6nOUR"S UUUUS .U D6n[+U UUS 9$)aK qformer_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the Q-Former. Input tokens can optionally be provided to serve as text prompt, which the Q-Former model will encode. Indices can be obtained using [`InstructBlipProcessor`]. See [`InstructBlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) qformer_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Only relevant in case an encoder-decoder language model (like T5) is used. )rrrNrnrr%rs)rrrrdrer&r&r)r&rrrrBr.)rrNonesrprrrrru ones_likerr}rrr9rrr masked_scatterrSuse_decoder_only_language_modelr:)rGrr rrrrrr&rrr, image_embedsimage_attention_maskrquery_attention_mask query_outputs query_outputlanguage_model_inputsr outputss r7rInstructBlipModel.forwardsXP** %%=   &a(  %zz,*;*;*=cr*B%**]i]p]pq((// 0B0B10Er2N $zz,*;*;*=cr*B%**]i]p]pq ! )%*__5F%G "!7!:!:;O;V;V!W!&,@+Y_`!a  '1%".#7    %Q'+A\->->q-A+A1(DE   //DDFyQM%!&!;!% 8 8 F 5 8 89M9M}ObOb c!66y6^%445GI^_ ;; 6 6))+-G ))+-"3'=   G?))#*  r6rrrrr)NNNNNNF)r/r0r1r2r;_keep_in_fp32_modulesr&rZrrNrrOr rrrrrr rJr:rr5rrs@r7rrusH %O+, 1 ?( "e.>.> "uO`O` " ;?.22659:>-1)._ ''_ !,,_ !& 0 04 7 _ $$t+ _ ((4/ _ !++d2_ !& 0 04 7_ ||d*_ #'_ -._  @ @_ _ r6ra InstructBLIP Model for generating text given an image and an optional text prompt. The model consists of a vision encoder, Querying Transformer (Q-Former) and a language model. One can optionally pass `input_ids` to the model, which serve as a text prompt, to make the language model continue the prompt. Otherwise, the language model starts generating text from the [BOS] (beginning-of-sequence) token. c^\rSrSr%\\S'SrSrS/rS\4U4Sjjr Sr S\ R4S jr SU4S jjrS rS r\\SS\R(S\R*S\R*S -S\S -S\\S\\-4 Sjj55rS\R*S\R(4Sjr\\SS\R(S\R(S\R*S -S\R(S -S\R*S -S\R*S -S\R*S -S\R(S -S\R*S -S\S\\S\\-4Sjj55r\R>"5S S\R(S\R*S -S\R*S -S\R*S -S\R*S -S\R(S -S\S\R*4Sjj5r Sr!U=r"$)!rirSrTrc>[TU]U5 [RUR5Ul[ R"[R"SURURR55Ul [RUR5Ul[ R "URRUR"R5UlUR&(a![(R*"UR"5nO [,R*"UR"5nX lUR15 gr)rYrZr* _from_configrrrr_rNrrrr[rrrrrrrr#rr$rr/)rGrSrrhs r7rZ-InstructBlipForConditionalGeneration.__init__"s  3@@AUAUVLLQ8O8OQWQfQfQrQr)st/<P>PQN2>>v?Q?QRN, r6c:URRU5 gr)rset_output_embeddings)rGnew_embeddingss r7r':InstructBlipForConditionalGeneration.set_output_embeddings6s 11.Ar6r?c6URR5$r)rget_output_embeddingsrLs r7r+:InstructBlipForConditionalGeneration.get_output_embeddings9s""88::r6NcX>UcURR5$[TU]US9$)N)modality)r get_encoderrY)rGr.rhs r7r/0InstructBlipForConditionalGeneration.get_encoder<s1  &&224 47&&9 9r6c6URR5$r)r get_decoderrLs r7r20InstructBlipForConditionalGeneration.get_decoderBs""..00r6c"URn[U5S:a=SU;a7[RR 5S:a[ R S5 [URS5(aSURRl ggrrrs r7r;InstructBlipForConditionalGeneration._preprocess_accelerateFrr6r rrrc UR"S UUSS.UD6n[S 0UDSU0D6nUSn[R"UR 5SS[R UR S9nURRURSSS5n [R"U R 5SS[R UR S9n Uc[R"U5nURU R 5n[R"X/SS 9nUR"S UUU UUSS .UD6n XlU SSS2SU R S52SS24n URU 5n XlU$) a pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. qformer_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the Q-Former. Input tokens can optionally be provided to serve as text prompt, which the Q-Former model will encode. Indices can be obtained using [`InstructBlipProcessor`]. See [`InstructBlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) qformer_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) T)rr return_dictr,rNrnrr%rs)rrrrdrer7r.)rr*rNrrprrrrrurrr}rr-rr4)rGrr rrrr,rrrrr-rimage_featuress r7get_image_features7InstructBlipForConditionalGeneration.get_image_featuresZs>6:5F5F6 %%=6  6  Aq>qbpq%a(  %zz,*;*;*=cr*B%**]i]p]pq((// 0B0B10Er2N $zz,*;*;*=cr*B%**]i]p]pq ! )%*__5F%G "!7!:!:;O;V;V!W!&,@+Y_`!a,, '1%".#7   *9&&q)!-C|/@/@/C-CQ*FG 11,?'5$r6rr&c dUcjX R5"[R"URR[R UR S95:HnURS5nOXRR:HnURS5RUR 5nU$rrrs r7r 9InstructBlipForConditionalGeneration.get_placeholder_maskr r6rrrlabelsc URUUUU SS9n U Rn U RnU RnUcUR 5"U5nUc[ R "U5nU RURUR5n URXHS9nURUU 5nURR(aSUR"S UUS.U D6nUSnSnU b3UR"S UXRR R"S.U D6nO5SU S'UR"S UUUUU S .U D6nUR$nUR&n[)UUUUUS 9$) a qformer_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the Q-Former. Input tokens can optionally be provided to serve as text prompt, which the Q-Former model will encode. Indices can be obtained using [`InstructBlipProcessor`]. See [`InstructBlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) qformer_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Only relevant in case an encoder-decoder language model (like T5) is used. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import InstructBlipProcessor, InstructBlipForConditionalGeneration >>> import torch >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> model = InstructBlipForConditionalGeneration.from_pretrained("Salesforce/instructblip-vicuna-7b") >>> processor = InstructBlipProcessor.from_pretrained("Salesforce/instructblip-vicuna-7b") >>> device = "cuda" if torch.cuda.is_available() else "cpu" >>> model.to(device) # doctest: +IGNORE_RESULT >>> url = "https://raw.githubusercontent.com/salesforce/LAVIS/main/docs/_static/Confusing-Pictures.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())).convert("RGB") >>> prompt = "What is unusual about this image?" >>> inputs = processor(images=image, text=prompt, return_tensors="pt").to(device) >>> outputs = model.generate( ... **inputs, ... do_sample=False, ... num_beams=5, ... max_length=256, ... min_length=1, ... top_p=0.9, ... repetition_penalty=1.5, ... length_penalty=1.0, ... temperature=1, ... ) >>> generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0].strip() >>> print(generated_text) The unusual aspect of this image is that a man is ironing clothes on the back of a yellow SUV, which is parked in the middle of a busy city street. This is an unconventional approach to ironing clothes, as it requires the man to balance himself and his ironing equipment on top of the vehicle while navigating through traffic. Additionally, the presence of taxis and other vehicles in the scene further emphasizes the unusual nature of this situation. ```Tr rrr7Nrrr)r=r=rr7)r&rrrr=)r<r=r,r-r>r.)r9r4r-r,r9rNrrrrr rrSrr loss_functionrrr<r=r:)rGrr rrrrrr&r=rrr8rr-r,r rr=r<s r7r,InstructBlipForConditionalGeneration.forwards^CGBYBY /#9%= CZC !/ < <(88'66   557 BM  !"__Y7N 5 8 89M9M}ObOb c!66y6^%445GI^_ ;; 6 6))+-G QZFD!))!&[[=T=T=_=_ci %)F= !))+-"3'=   G<)+#*   r6c T[US5(aUR5 URSn URUUUUSS9n U Rn UcUcUR R /UR R-n XR RR/-n [R"U /[RURS9nURU S5nUR5"U5nUc[R "U5nU R#URUR$5n UR'XFS9nUR)X5nXeS.nUR*R R,(dXOS 'UR*R."S 0UDUD6nU$) a Overrides `generate` function to be able to use the model as a conditional generator. Args: pixel_values (`torch.FloatTensor` of shape (batch_size, num_channels, height, width)): Input images to be processed. qformer_input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt to be fed to the Q-Former module. qformer_attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt for the generation. attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Embedded representation of the inputs. Should be float, not int tokens. interpolate_pos_encoding (`bool`, *optional*, defaults to `False`): Whether to interpolate the positional encoding of the image embeddings. Returns: captions (list): A list of strings of length batch_size * num_captions. rrTr?rr%rrrr.)r@rrur9r4rSimage_token_indexrr bos_token_idrNrrrrepeatr9rrrr rris_encoder_decodergenerate)rGrr rrrr&rgenerate_kwargsrr8r image_tokens start_tokensr inputsrs r7rG-InstructBlipForConditionalGeneration.generate3sD 4 ) )  ' ' )!''* BFBYBY /#9%= CZC !/ < <   $ = =>A]A]] +{{/F/F/S/S.TT !LL,uzzR^ReRef %,,Z;  557 BM  !"__Y7N 5 8 89M9M}ObOb c!66y6^%445G_ #0S""))<<"+; %%..KK?Kr6r rr)NNNNNNNF)NNNNNF)#r/r0r1r2r&r4r;rr!rZr'rModuler+r/r2rrrrNrOrrrrrJr*r9r r:rrrGr5rrs@r7rrs $O!+,1(B;ryy;: 1?( ;?05 @''@!++@!& 0 04 7 @ #'+ @ +, @ 8 8@@D "e.>.> "uO`O` " ;?.22659:>26*.).B ''B !,,B !& 0 04 7 B $$t+ B ((4/ B !++d2B !& 0 04 7B ((4/B   4'B #'B +,B  @ @B B H ]]_6::>-12626).D''D!++d2D!& 0 04 7 D ##d* D ((4/ D((4/D#'D   DDr6r)rrrrr*)r)Ur3r`collections.abcr dataclassesrtypingrrNrrr activationsr generationr masking_utilsr modeling_flash_attention_utilsr modeling_layersr modeling_outputsrrrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsrrrrrr utils.genericrutils.output_capturingrr autor"r#r$configuration_instructblipr&r'r( get_loggerr/rr*r:rMrQrfloatrrrrrrr*rrrrrrrrrrr__all__r.r6r7rcs" $! &!)6B9G&6jj7EIIoo   H %  P.H P  P   k