3j?|SrSSKJr SSKJr SSKrSSKJr SSKJr SSK J r SS K J r SS KJr SS KJr SS KJrJrJr SS KJrJr SSKJr SSKJr SSKJrJrJ r J!r!J"r" SSK#J$r$ SSK%J&r& SSK'J(r( \!RR"\*5r+"SS\RX5r-S?S\RXS\R\S\R\S\R\S\R\S-S\/S\/4Sjjr0"SS \RX5r1"S!S"\RX5r2"S#S$\RX5r3"S%S&\RX5r4"S'S(\RX5r5"S)S*\5r6"S+S,\RX5r7\"S-S.\55r8\"S/S0\855r9"S1S2\RX5r:"S3S4\RX5r;\"S5S6\855r<\"S7S89\"S9S:\555r=\"S;S89"S<S=\855r>/S>Qr?g)@zPyTorch Splinter model.)Callable) dataclassN)nn)CrossEntropyLoss)initialization)ACT2FN)create_bidirectional_mask)GradientCheckpointingLayer)BaseModelOutput ModelOutputQuestionAnsweringModelOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward)TransformersKwargsauto_docstringcan_return_tupleloggingtorch_compilable_check)merge_with_config_defaults)capture_outputs)SplinterConfigc ^\rSrSrSrU4SjrS S\RS-S\RS-S\RS-S\RS-S \ 4 S jjr S r U=r $) SplinterEmbeddings(zGConstruct the embeddings from word, position and token_type embeddings.cv>[TU]5 [R"URUR UR S9Ul[R"URUR 5Ul [R"URUR 5Ul [R"UR URS9Ul [R"UR5UlUR#S[$R&"UR5R)S5SS9 g)N) padding_idxeps position_idsrF) persistent)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandselfconfig __class__s h/home/wildlama/miniconda3/lib/python3.13/site-packages/transformers/models/splinter/modeling_splinter.pyr(SplinterEmbeddings.__init__+s !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej  N input_idstoken_type_idsr# inputs_embedsreturncUbUR5nOUR5SSnUSnUcURSS2SU24nUc8[R"U[RURR S9nUcUR U5nURU5nXG-nURU5n X- nURU5nURU5nU$)Nr%rdtypedevice) sizer#r8zeroslongrIr-r1r/r2r6) r<rBrCr#rD input_shape seq_lengthr1 embeddingsr/s r?forwardSplinterEmbeddings.forward9s  #..*K',,.s3K ^  ,,Q  ^ J": "66|D) ^^J/ \\*- rA)r2r6r/r1r-)NNNN) __name__ __module__ __qualname____firstlineno____doc__r(r8 LongTensor FloatTensortuplerP__static_attributes__ __classcell__r>s@r?rr(sQ  .2260426 ##d*((4/&&-  ((4/   rArmodulequerykeyvalueattention_maskscalingr6c [R"XRSS55U-nUbX-n[RR US[R S9RUR5n[RRXURS9n[R"X5n U RSS5R5n X4$)Nrr%)dimrH)ptrainingr) r8matmul transposer functionalsoftmaxfloat32torHr6rg contiguous) r]r^r_r`rarbr6kwargs attn_weights attn_outputs r?eager_attention_forwardrr[s<<}}Q':;gEL!#4 ==((2U]](SVVW\WbWbcL==((6??([L,,|3K''1-88:K  $$rAc ^\rSrSrU4SjrS S\R S\RS-S\\ S\ \R \R S-44Sjjr S r U=r $) SplinterSelfAttentionrc6>[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35eXlURUl[URUR- 5UlURUR-Ul [R"URUR5Ul [R"URUR5Ul [R"URUR5Ul[R"UR 5UlUR UlURS-Ulg)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()g)r'r(r+num_attention_headshasattr ValueErrorr=intattention_head_size all_head_sizerLinearr^r_r`r4attention_probs_dropout_probr6attention_dropoutrbr;s r?r(SplinterSelfAttention.__init__ssD     : : :a ?PVXhHiHi#F$6$6#78 445Q8   #)#=#= #&v'9'9F[TU]5 [R"URUR5Ul[R "URURS9Ul[R"UR5Ul gNr!) r'r(rrr+denser2r3r4r5r6r;s r?r(SplinterSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= rAr input_tensorrEcpURU5nURU5nURX-5nU$rrr6r2r<rrs r?rPSplinterSelfOutput.forward5 =1  ]3 }'CD rAr2rr6 rRrSrTrUr(r8rrPrZr[r\s@r?rr6> U\\RWR^R^rArc ^\rSrSrU4SjrS S\R S\RS-S\\ S\R 4Sjjr S r U=r $) SplinterAttentioncb>[TU]5 [U5Ul[ U5Ulgr)r'r(rtr<routputr;s r?r(SplinterAttention.__init__s& )&1 (0 rANrrarorEc ZUnUR"U4SU0UD6upURX5nU$Nra)r<r)r<rraroresidual_s r?rPSplinterAttention.forwardsE !99  )    M< rA)rr<r) rRrSrTrUr(r8rrXrrrPrZr[r\s@r?rrsV148 || ))D0 +,    rArcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)SplinterIntermediatec>[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl gr) r'r(rrr+intermediate_sizer isinstance hidden_actstrr intermediate_act_fnr;s r?r(SplinterIntermediate.__init__s` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $rArrEcJURU5nURU5nU$rrr)r<rs r?rPSplinterIntermediate.forwards& =1 00? rArrr\s@r?rrs(9U\\ellrArcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $)SplinterOutputc(>[TU]5 [R"URUR 5Ul[R"UR URS9Ul[R"UR5Ul gr) r'r(rrrr+rr2r3r4r5r6r;s r?r(SplinterOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= rArrrEcpURU5nURU5nURX-5nU$rrrs r?rPSplinterOutput.forwardrrArrr\s@r?rrrrArc ^\rSrSrU4SjrS S\R S\RS-S\\ S\R 4Sjjr S r S r U=r $) SplinterLayerc>[TU]5 URUlSUl[ U5Ul[ U5Ul[U5Ul g)Nr) r'r(chunk_size_feed_forward seq_len_dimr attentionr intermediaterrr;s r?r(SplinterLayer.__init__sI '-'E'E$*6208$V, rANrrarorEc UR"U4SU0UD6n[URURURU5nU$r)rrfeed_forward_chunkrr)r<rraros r?rPSplinterLayer.forwardsW   )  2  # #T%A%A4CSCSUb rAcJURU5nURX!5nU$r)rr)r<attention_outputintermediate_output layer_outputs r?r SplinterLayer.feed_forward_chunk s)"//0@A{{#6I rA)rrrrrr)rRrSrTrUr(r8rrXrrrPrrZr[r\s@r?rrs[-48||))D0+,   $rArc |^\rSrSrU4SjrS S\R S\RS-S\\ S\ 4Sjjr S r U=r $) SplinterEncoderic>[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5UlSUl gs snf)NF) r'r(r=r ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r<r=ir>s r?r(SplinterEncoder.__init__sR  ]]5IaIaCb#cCbaM&$9Cb#cd &+#$dsA&NrrarorEc NURH nU"UU40UD6nM [US9$)Nlast_hidden_state)rr )r<rraro layer_modules r?rPSplinterEncoder.forwards> !JJL(M'+  rA)r=rrr)rRrSrTrUr(r8rrXrrr rPrZr[r\s@r?rrsR,48 || ))D0 +,     rArcF^\rSrSr%\\S'SrSr\\ S.r U4Sjr Sr U=r $)SplinterPreTrainedModeli)r=splinterT)r attentionsc>[TU]U5 [U[5(a\[R "UR [R"UR RS5RS55 gg)Nr%r$) r' _init_weightsrrinitcopy_r#r8r9rr:)r<r]r>s r?r%SplinterPreTrainedModel._init_weights3s^ f% f0 1 1 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h 2rA)rRrSrTrUr__annotations__base_model_prefixsupports_gradient_checkpointingrrt_can_record_outputsrrZr[r\s@r?rr)s/ "&*#&+ iirArc^\rSrSrSrU4SjrSrSr\\ \ SS\ RS-S\ RS-S \ RS-S \ RS-S \ RS-S \ \S \\-4Sjj555rSrU=r$) SplinterModeli9a" The model is an encoder (with only self-attention) following the architecture described in [Attention is all you need](https://huggingface.co/papers/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. c>[TU]U5 Xl[U5Ul[ U5UlUR5 gr)r'r(r=rrOrencoder post_initr;s r?r(SplinterModel.__init__As9   ,V4&v.  rAc.URR$rrOr-)r<s r?get_input_embeddings"SplinterModel.get_input_embeddingsKs...rAc$XRlgrr)r<r`s r?set_input_embeddings"SplinterModel.set_input_embeddingsNs */'rANrBrarCr#rDrorEc UbUb [S5eUb"URX5 UR5nO"UbUR5SSnO [S5eUupUb URO URn Uc[R "X4U S9nUc$[R "U[RU S9nURUUUUS9n [URU US9nUR"U 4S U0UD6n U S n [U S 9$) a token_type_ids (`torch.LongTensor` of shape `batch_size, sequence_length`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `batch_size, sequence_length`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) NzDYou cannot specify both input_ids and inputs_embeds at the same timer%z5You have to specify either input_ids or inputs_embeds)rIrG)rBr#rCrD)r=rDrararr) r{%warn_if_padding_and_no_attention_maskrJrIr8onesrKrLrOr r=rr )r<rBrarCr#rDrorM batch_sizerNrIembedding_outputencoder_outputssequence_outputs r?rPSplinterModel.forwardQs16  ]%>cd d  "  6 6y Q#..*K  &',,.s3KTU U!, %.%:!! @T@T  !"ZZ*)A6RN  !"[[EJJvVN??%)' + 3;;*)  ,,  )   *!,-  rA)r=rOr)NNNNN)rRrSrTrUrVr(rrrrrr8rrrrYr rPrZr[r\s@r?rr9s /0 *..2.2,0-1 @ <<$&@  t+@  t+ @ llT) @ ||d* @ +,@   @  @ rArcf^\rSrSrSU4SjjrS\R S\R 4SjrSrU=r $)SplinterFullyConnectedLayeric>[TU]5 XlX l[R "URUR5Ul[UUl[R"UR5Ul gr) r'r( input_dim output_dimrrrr act_fnr2)r<rr rr>s r?r($SplinterFullyConnectedLayer.__init__sR "$YYt~~t? Z( doo6rAinputsrEclURU5nURU5nURU5nU$r)rr r2)r<r rs r?rP#SplinterFullyConnectedLayer.forwards2 6*  M2 }5 rA)r2r rrr )gelurr\s@r?rrs(7ellu||rArc2^\rSrSrSrU4SjrSrSrU=r$)QuestionAwareSpanSelectionHeadiz^ Implementation of Question-Aware Span Selection (QASS) head, described in Splinter's paper: c>[TU]5 [URUR5Ul[URUR5Ul[URUR5Ul[URUR5Ul[R"URURSS9Ul [R"URURSS9Ul g)NF)bias) r'r(rr+query_start_transformquery_end_transformstart_transform end_transformrrstart_classifierend_classifierr;s r?r('QuestionAwareSpanSelectionHead.__init__s %@ASASU[UgUg%h"#>v?Q?QSYSeSe#f :6;M;MvOaOab89K9KVM_M_` " &*<*P>PW\ ] ii(:(:F? ''1a0 ||M> ++N; ##Aq!,\\-: ''rA)rrrrrr) rRrSrTrUrVr(rPrZr[r\s@r?rrs \((rArc^^\rSrSrU4Sjr\\SS\RS-S\RS-S\RS-S\RS-S\RS-S \RS-S \RS-S \RS-S \ \ S \ \ -4Sjj55rSrU=r$)SplinterForQuestionAnsweringic>[TU]U5 [U5Ul[ U5UlUR UlUR5 grr'r(rrr splinter_qassquestion_token_idrr;s r?r(%SplinterForQuestionAnswering.__init__C  %f- ;FC!'!9!9 rANrBrarCr#rDstart_positions end_positionsquestion_positionsrorEc  Sn UcUbB[R"[R"XR5R 5SS9n OH[R "UR S5[RURURS9n U RS5nSn UR"U4UUUUS.U D6n U Sn URX5upU (a!URS 5URS 5pUbfUS U- [R"UR5R --nUS U- [R"UR5R --nSnUbUb[#UR 55S :aURS5n[#UR 55S :aURS5nUR S 5nUR%SU5 UR%SU5 ['US 9nU"X5nU"X5nUU-S - n[)UUUU R*U R,S 9$) aI token_type_ids (`torch.LongTensor` of shape `batch_size, sequence_length`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `batch_size, sequence_length`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) question_positions (`torch.LongTensor` of shape `(batch_size, num_questions)`, *optional*): The positions of all question tokens. If given, start_logits and end_logits will be of shape `(batch_size, num_questions, sequence_length)`. If None, the first question token in each sequence in the batch will be the only one for which start_logits and end_logits are calculated and they will be of shape `(batch_size, sequence_length)`. FNr%)rer)rHlayoutrITrarCr#rDr ignore_indexrdlossr'r(rr)r8argmaxeqr/r|rKrJrLr6rIrrr.squeezefinforHminlenclamp_rrrr)r<rBrarCr#rDr2r3r4roquestion_positions_were_none"question_position_for_each_exampleoutputsrr'r( total_loss ignored_indexloss_fct start_lossend_losss r?rP$SplinterForQuestionAnswering.forwards7D(-$  %$5:\\XXi)?)?@EEGR626;[[!&&q)MDXDXanauau62"D!M!Mb!Q +/ (--  ))%'    "!*#'#5#5o#Z ''3';';A'> @R@RST@U*  %'1~+=\M_M_A`AdAd*ddL#q>'9U[[IYIY=Z=^=^&^^J  &=+D?'')*Q."1"9"9""==%%'(1, - 5 5b 9 (--a0M  " "1m 4  M 2']CH!,@J :H$x/14J+%!!//))   rAr/rr.NNNNNNNN)rRrSrTrUr(rrr8rrWrrrYrrPrZr[r\s@r?r+r+s*..2.2,0-137156:W <<$&W  t+W  t+ W llT) W ||d* W ))D0W ''$.W ",,t3W +,W  - -W W rAr+zB Class for outputs of Splinter as a span selection model. ) custom_introc\rSrSr%SrSr\RS-\S'Sr \RS-\S'Sr \RS-\S'Sr \ \RS-\S'Sr \ \RS-\S'S rg) SplinterForPreTrainingOutputi7a loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when start and end positions are provided): Total span extraction loss is the sum of a Cross-Entropy for the start and end positions. start_logits (`torch.FloatTensor` of shape `(batch_size, num_questions, sequence_length)`): Span-start scores (before SoftMax). end_logits (`torch.FloatTensor` of shape `(batch_size, num_questions, sequence_length)`): Span-end scores (before SoftMax). Nr;r'r(rrr)rRrSrTrUrVr;r8rXrr'r(rrYrrZrrAr?rPrP7s|&*D%  d ")-1L%##d*1+/J!!D(/59M5**+d2926Je''(4/6rArPz Splinter Model for the recurring span selection task as done during the pretraining. The difference to the QA task is that we do not have a question, but multiple question tokens that replace the occurrences of recurring spans instead. c^\rSrSrU4Sjr\\SS\RS-S\RS-S\RS-S\RS-S\RS-S \RS-S \RS-S \RS-S \ \ S \ \ -4Sjj55rS\RS \R4SjrSrU=r$)SplinterForPreTrainingiNc>[TU]U5 [U5Ul[ U5UlUR UlUR5 grr-r;s r?r(SplinterForPreTraining.__init__Vr1rANrBrarCr#rDr2r3r4rorEc  UcUbUb [S5eUcUc [S5eUcURU5nUR"U4UUUUS.U D6n U Sn U R5upnUR X5unnURS5nUbUR S5R U UU 5nUSU- [R"UR5R--nUSU- [R"UR5R--nSnUbUbURS[SU S- 55 URS[SU S- 55 [URRS9nU"UR!U U-U 5UR!U U-55nU"UR!U U-U 5UR!U U-55nUU-S- n[#UUUU R$U R&S 9$) a input_ids (`torch.LongTensor` of shape `(batch_size, num_questions, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `batch_size, num_questions, sequence_length`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `batch_size, num_questions, sequence_length`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_questions, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. start_positions (`torch.LongTensor` of shape `(batch_size, num_questions)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size, num_questions)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. question_positions (`torch.LongTensor` of shape `(batch_size, num_questions)`, *optional*): The positions of all question tokens. If given, start_logits and end_logits will be of shape `(batch_size, num_questions, sequence_length)`. If None, the first question token in each sequence in the batch will be the only one for which start_logits and end_logits are calculated and they will be of shape `(batch_size, sequence_length)`. NzCquestion_positions must be specified in order to calculate the lossz?question_positions must be specified when inputs_embeds is usedr7rrr8rdr:) TypeError_prepare_question_positionsrrJr.rr:r8r?rHr@rBmaxrr=r,rrPrr)r<rBrarCr#rDr2r3r4rorErrsequence_lengthrer'r( num_questions attention_mask_for_each_questionrFrHrIrJs r?rPSplinterForPreTraining.forward`s=j  %/*E-Jcab b  'I,=]^ ^  '!%!A!A)!L --  ))%'    "!*+:+?+?+A( S#'#5#5o#Z j*//2  %/=/G/G/J/Q/QM?0 ,(1/O+OSXS^S^_k_q_qSrSvSv*vvL#q+K'Ku{{[e[k[kOlOpOp&ppJ  &=+D  " "1c!_q-@&A B  C?Q+>$? @ (T[[5M5MNH!!!*}"H%x/14J+%!!//))   rAc6[R"XRR:H5up#[R"U5n[R "UR S5UR54URR[RURS9n[UR S5UR S5:HS5 [R"UVs/sHn[R"U5PM sn5nX5X'4'U$s snf)NrrGz?All samples in the batch must have at least one question token.)r8wherer=r/bincountfullrJrXr,rLrIrcatr9)r<rBrowsflat_positionsrZr!ncolss r?rW2SplinterForPreTraining._prepare_question_positionss${{9 8U8U+UVt, JJ ^^A  1 1 3 4 KK $ $**##     q !Y^^A%6 6 M yy=A=a%,,q/=AB .$*Bs( DrLrM)rRrSrTrUr(rrr8rrWrrrYrPrPrWrZr[r\s@r?rRrRNs'*..2.2,0-137156:n <<$&n  t+n  t+ n llT) n ||d* n ))D0n ''$.n ",,t3n +,n  - -n n `U\\ellrArR)r+rRrrr)r)@rVcollections.abcr dataclassesrr8rtorch.nnrrr activationsr masking_utilsr modeling_layersr modeling_outputsr r rmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsrrrrr utils.genericrutils.output_capturingrconfiguration_splinterr get_loggerrRloggerModulerrfloatrrrtrrrrrrrrrrr+rPrR__all__rrAr?r{s$! %&!69ZZF&6jj752   H %//t% II% <<% % << % LL4' %  %%.3)BII3)n   . 299  RYY .D bii 2 io i iZ +Z Z z"))$#(RYY#(Ld #:d d N  7;7  7"L4LL^ rA