3jSrSSKrSSKrSSKrSSKJr SSKJr SSKJ r SSK J r SSK J r SSKJr SSKJrJr SS KJr SS KJr SS KJr SS KJrJr SS KJrJ r J!r!J"r" SSK#J$r$ SSK%J&r& SSK'J(r( SSK)J*r* SSK+J,r,J-r-J.r.J/r/J0r0J1r1J2r2 SSK3J4r4J5r5J6r6 \6Rn"\85r9Sr:Sr;Sr\,S- r,\\ \!\"S.r?\:\=S.r@\5"\,5"SS\/55rA\ArBg)z Tokenization classes for fast tokenizers (provided by HuggingFace's tokenizers library). For slow (python) tokenizers see tokenization_utils.py N) defaultdict)Iterable)copyfile)Any)is_offline_mode) AddedToken processors)Encoding) Tokenizer)Decoder)BPEUnigram) BpeTrainerUnigramTrainerWordLevelTrainerWordPieceTrainer) cached_file) SpmConverter)convert_gguf_tokenizer)load_gguf_checkpoint)INIT_TOKENIZER_DOCSTRING BatchEncodingPreTokenizedInputPreTrainedTokenizerBase TextInputTruncationStrategygenerate_merges)PaddingStrategyadd_end_docstringsloggingztokenizer.jsonzspecial_tokens_map.jsonztokenizer_config.jsonztokenizer.modelzadded_tokens.jsonu tokenizer_object ([`tokenizers.Tokenizer`]): A [`tokenizers.Tokenizer`] object from 🤗 tokenizers to instantiate from. See [Using tokenizers from 🤗 tokenizers](../fast_tokenizers) for more information. tokenizer_file ([`str`]): A path to a local JSON file representing a previously serialized [`tokenizers.Tokenizer`] object from 🤗 tokenizers. )r r WordLevel WordPiece)tokenizer_file vocab_filec)^^\rSrSrSr\rSrSr\ SNSj5r U4Sjr \ S\ 4Sj5r\ S\ 4S j5rSOS \S \S-S\\4S jjrS r\ S5r\ S5r\R,S5r\R,S5rSr\ S\4Sj5rS\\\44Sjr\ S\\\44Sj5r\ S\\\44Sj5r\ S\\\44Sj5r\r \r!S\\\44Sjr"S\ 4Sjr#S\4Sjr$\ S\%4Sj5r&\ S\'4Sj5r(SPS\)S\ S-S \ S-S!\ S"\ S#\ S$\ S%\ S\\\\*4\+\)44S&jjr,S'\S\4S(jr-S)\S\S-4S*jr.SNS+\+\\-S\4S,jjr/SNS-\ S\4S.jjr0SNS/\\+\-S0\ S\\+\-4S1jjr1SQS2\S-\S-S3\ S\+\4S4jjr2S5\3S6\4S7\S8\S9\S-S:\S-4 S;jr5SS\3Rl\4RnSS\ S9\S-S:\S-S?\ S-S\ S-S \ S-S!\ S"\ S#\ S$\ S%\ S@\ S-S\:4(SAjjr;SB\+\S\4SCjr<SRSD\\+\-S0\ SE\ S-S\4SFjjr=SSS \\>R~-SG\\SH4SI\ S-S \S-S\\SH44 SJjjr@STSKjrA\ SUSLj5rBSMrCU=rD$)VTokenizersBackendSa5 Base class for all fast tokenizers (wrapping HuggingFace tokenizers library). Inherits from [`~tokenization_utils_base.PreTrainedTokenizerBase`]. Handles all the shared methods for tokenization and special tokens, as well as methods for downloading/caching/loading pretrained tokenizers, as well as adding tokens to the vocabulary. This class also contains the added tokens in a unified way on top of all tokenizers so we don't have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...). NFc "^([U5nURSS5nUb_[RR U5(a;U[ LdSUR ;dU(a[R"U5US'U$UGb[RR U5(Ga[USS9n[R"U5nSSS5 WRS05RS5nUS ;a][U5n[US5n 0U S 'US :Xa/U S 'XS'/US '[R"[R"U55n O[R"U5n U RUS'U R US'U R"US'U R"bU R"US'U R bU R US'URS5n U (awU RSS5S:XaU Sn O[%U [&5(dU /n U H=n U RS5S:XdMSU ;dM"SSKn U R+U S5US' O URS05RS S5nUR,c0[%U[&5(a['[/[0U55nGO0UR,R2S:XaV[%U[&5(a@U(a9[%US[&[045(aUVs/sHn[1U5PM nnOUR,R2S:Xa"[5U5VVs0sH unnUU_M nnnOUR,R2S :XdUR,R2S:XaP[%U[&5(a;[5U5VVs0sH#unn[%U[&5(aUSOUU_M% nnnXS '[7USS5nS URS05;anU(agUR2S :XaWUSS nUVs/sH=n[%U[85(a[1UR;S55O [1U5PM? nnUUS 'U$URS5nURS5nURS 5nURS 5n[%U[85(a`UR=S 5(aJ[RR U5(a&S!S"KJ n U"US#9RCU5uUS 'US 'U$[%U[85(Ga[RR U5(GaUR=S$5(GaxS!S%KJ"n U"U5nURF"UR,40UD6nS!S&KJ$n URUR25nUb#[KUS'5(aURL"S?0UD6n[KUS+5(aURT"S?0UD6nSU;GaU[ LdSUR ;GaURS S5nURS S5nURS,5=(d 0nUbU(aURW5VVs0sH unnUU_M nnnURW5H_unn[YU5n[9U5nURU5n U (dM6U U:wdM>UU;dMFURU 5UU'UUU'Ma [ZR\"UR^UUS-9n!U!bU!US'UR^R`n"U"RbS:a%UReS.U"Rf=(d S/5 U"RhS:a%UReS0U"Rj=(d S15 U"RlS:a%UReS2U"Rn=(d S35 U$UcC[%U[85(a.[RR U5(a UUS 'US nUcC[%U[85(a.[RR U5(a UUS 'US nUcUR,bUR,R2S :Xa[%U[5(akS:[t[vS;[&[84U(4S<jjm(/S=Qn%[y5n&U%H&n'U'U;dM U&R{T("UU'/55 M( [}UU&S>9nUUS 'U$!,(df  GN~=fs snfs snnfs snnfs snf![Na1nRSS(UR2S)US*35 SnAGNSnAff=fs snnf![NaVnRSS4US5US635 S!S7KJ8n# U#"UURS85S99n$U$Rs5US'SnAU$SnAff=f)@z Build a `tokenizers.Tokenizer` backend from the available serialization files (tokenizer.json, sentencepiece models, tekken.json, vocab/merges). r$N__init__tokenizer_objectutf-8encodingmodeltype)Nrvocabr merges added_tokenspost_processortokenizer_paddingtokenizer_truncation_json_truncation _json_padding normalizerSequence normalizers Precompiledprecompiled_charsmapr_spm_precompiled_charsmaprr"r# r% merges_filez tekken.jsonr)MistralConverter)r%.model)SentencePieceExtractor)SLOW_TO_FAST_CONVERTERSconvert_from_spmz,Could not reorder vocab using converter for z due to z/. Falling back to raw SentencePiece extraction.convert_from_spm_modeladded_tokens_decoder)protor1r2 bos_token eos_token unk_tokenzz+Could not extract SentencePiece model from z$ using sentencepiece library due to z%. Falling back to TikToken extractor.)TikTokenConverterextra_special_tokens)r%rOvaluesreturnc>/nUHVnUcM[U[[45(aURT"U55 M<UR [ U55 MX U$N) isinstancelisttupleextendappendstr)rP collectedval_iter_special_tokenss d/home/wildlama/miniconda3/lib/python3.13/site-packages/transformers/tokenization_utils_tokenizers.pyr\HTokenizersBackend.convert_to_native_format.._iter_special_tokens)sY') !C{ !#e}55!(()=c)BC!((S2 "! ) pad_tokenrMrIrK sep_token cls_token mask_tokenadditional_special_tokensrO) skip_tokens)?dictpopospathisfiler'__dict__ TokenizerFast from_fileopenjsonloadgetfrom_strdumpsr4padding truncationrTrUbase64 b64decoder/maprV__name__ enumerategetattrrYsplitendswithconvert_slow_tokenizerrAextract_vocab_merges_from_modelrCextractrDhasattrrE ExceptionloggerwarningrFitemsintrbuild_tokenizer_from_spm_protorH trainer_specbos_id setdefault bos_pieceeos_id eos_pieceunk_id unk_piecerN convertedrrsetupdater))clstrust_remote_codekwargs local_kwargsfast_tokenizer_filetokenizer_handletokenizer_json model_typeminimal_tokenizer_json minimal_model tok_from_filenormalizer_configr9rwr1itemitokenr2merger%r@rArC extractorrDconverter_classerGtoken_id id_to_token new_token current_tokenr+ proto_specrN converterspecial_tokens_keysrekeyr\s) @r]convert_to_native_format*TokenizersBackend.convert_to_native_formatesH F| *../?F +233))Zs||-KO`/CY]gCg%DJDTDT&'=>E %@A#4#&&w377FEyy eT** UE!23E##y0eT**uE!HtUZm9\9\5:;UTU4[UE;##{22;E2BC2Bha2BC##u, 0B0Bk0QeT**_hin_op_oS[STV[E4)@)@U1XeQN_oEp$) ! gt4J>--gr:: zObObfkOk'0:kqrkqbgZs5K5K% C 01QVW\Q]]kqr)/ X& !%%l3 "&&}5   )!!(+ j# & &:+>+>}+M+MRTRYRYR`R`akRlRl @>*+E+E*J]J]^fJgJgF IJ3:> (00KlK  O&=&A&A#,,&OO&2wPb7c7c'6'G'G'W,'W 3 899#&#=#=#M #ML &\9,, #,,0N(,,Wd;E)--h=F,8+;+;jc::rww~~k?Z?Z%0L "!(+F >cii3 8J8Je8SXbchjnXoXo !Xc] !tCy ! # %(EK*,&&&';\#=Nf.f.C:f. e* f%4&f f. f%% f.. h8A h  hc <>URSS5nURSS5nURSS5 URSS5nURSS5nURSS5nURS05nURSS 5n URS 5n URS 5n URS 5n Sn Ub[R"U5n GOUb<[R R U5(a[R"U5n GOUb|[URS S5U40UD6n[U5nUSSnUSnUSn[UU5un nURU5 [U5S:aURU5 GOURcU bU bT[U [ 5(aU O&[#U 5VVVs0sH ununnUU_M snnnn[[%UU SSS95n O[U [ 5(a[[%U /SSS95n Ou[U [&5(aGU (a@[U S[([&45(a"[[+XRSS5S95n OURc [-S5eUc4Uc1URc$UR/SS5 UR/SS5 U bXlURc [-S5eURSS5=(d URR0=(d UnUbqURR2"S80UD6 UR/SUS5 UR/S US!5 UR/S"US"5 UR/S#US$5 OURR55 URS%S5=(d URR6=(d UnUbURR8"S80UD6 UR/S&US&5 UR/S'US(5 UR/S)US!5 UR/SUS*5 UR/S+US+5 S,U;aS-US,'S.U;=(d S/U;nURS.S 5UlURS/S 5UlURS0S5=n(aUURlU=(d URR>SLUl [BT&U]"S80UD6 U bXl#Xl$URJURl&URNVs1sHn[Q[SU55iM nn[UURW5S1S29VVs/sH"unn[Q[SU55U;dM UPM$ nnn['URXR[55UVs/sHn[]U5PM sn-n UR^Ra5H2n!U!cM[]U!5U ;dMU!U;dM!URcU!5 M4 URdH-n[]U5U ;dMUU;dMURcU5 M/ [U5S:a/n"UR^Ra5V#s/sHn#U#(dM []U#5PM n$n#UHpn[U[\5(a [gUSS39nO<[U[f5(a'URh(d[]U5U$;aSUl4U"RcU5 Mr U"(aURkU"5 URRm5n%U%S4:a[qURS5S5blURSS5 URr"URURtRS S54URtURS6S5S7.UD6UlUR@=(d URR>SLUl UR@(aURw5 ggs snnnfs snfs snnfs snfs sn#f![na Sn%GNf=f)9Nr7r8r>r+ gguf_filer$rGadd_prefix_spaceFr%r1r2 name_or_pathconfigr tokenizertokenizer_configrT)r1r2fuse_unkdropoutr)r1ra9Couldn't instantiate the backend tokenizer from one of: (1) a `tokenizers` library serialization file, (2) a slow tokenizer instance to convert or (3) an equivalent slow tokenizer class to instantiate and convert. You need to have sentencepiece or tiktoken installed to convert a slow tokenizer to a fast one.rIrJrKrLz3The backend tokenizer is not correctly initialized.r6 max_lengthtruncation_side directionstridetruncation_strategystrategyr5r`pad_token_type_id pad_type_id padding_sidelengthpad_to_multiple_ofbackend tokenizers add_bos_token add_eos_tokenr4c US$Nrrf)xs r],TokenizersBackend.__init__..s STUVSWr_r)speciali pre_tokenizerfix_mistral_regex) init_kwargsrrf)p$//B\B\p`p  " OO - - < <   lK ,E F   / J   h H(= >   3[5L M OO ) ) +::148dDOO ?   2H=Q4R S F " ,F9 %4%>%[/U[B["$jj%@$jj%@#ZZ(8$? ?> ?-;DOO *-G-q4??KiKimqKq* "6"  !(O 0040I0I-DHD]D]$^D]5T$u+%6D]!$^!'';'A'A'C X X uDK (AA  X  t005578Ta;bTa5CJTa;bb$(#;#;#B#B#D "*&'w6;NVc;c$$%89 $E//E5z(U--G$$U+0 }  !F040H0H0O0O0QW0Q1UVA0Q W&eS))&ud;Ez22 ==SZ;K-K(,  e$'' 779J  74??OT#R#^ JJ{D )"77  $$^T:!,,"(**-@$"G   DO  . . X$//2P2PTX2X *  - -  & & ( .oDl~%_ TokenizersBackend.get_vocab]s((4(@@r_c"UR5$rS)r>rs r]r1TokenizersBackend.vocab`s~~r_c[URR5SS9VVs0sHupURU_M snn$s snnf)z Returns the sorted mapping from string to index. The added tokens encoder is cached for performance optimisation in `self._added_tokens_encoder` for the slow tokenizers. c US$rrfrs r]r8TokenizersBackend.added_tokens_encoder..j dhijdkr_rrrGrcontentrvks r]r&TokenizersBackend.added_tokens_encoderds? *00I0I0O0O0QWk)lm)l 1 )lmmmAc6URR5$)z Returns the added tokens in the vocabulary as a dictionary of index to AddedToken. Returns: `dict[str, int]`: The added tokens. )rget_added_tokens_decoderrs r]rG&TokenizersBackend.added_tokens_decoderls7799r_c[URR5SS9VVs0sHupURU_M snn$s snnf)z Returns the added tokens in the vocabulary as a dictionary of token to index. Returns: `dict[str, int]`: The added tokens. c US$rrfrDs r]r3TokenizersBackend.get_added_vocab..rFr_rrGrIs r]get_added_vocab!TokenizersBackend.get_added_vocab{s?*00I0I0O0O0QWk)lm)l 1 )lmmmrMcg)z> Returns True, to avoid expensive `assert tokenizer` gotchas. Trfrs r]__bool__TokenizersBackend.__bool__sr_c4URRSS9$)z4 Size of the full vocabulary with the added tokens. Tr9r;rs r]__len__TokenizersBackend.__len__s---EEr_cUR$)zS `tokenizers.implementations.BaseTokenizer`: The Rust tokenizer used as a backend. )rrs r]backend_tokenizer#TokenizersBackend.backend_tokenizers r_c.URR$)zE `tokenizers.decoders.Decoder`: The Rust decoder for this tokenizer. )rdecoderrs r]r`TokenizersBackend.decoders &&&r_Tr.return_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverbosec UcSUR;nUcSUR;nU(aURbU/UR-n OU/n [[5n U Hn U SR U R 5 U(aU SR U R 5 U(aU SR U R5 U(aU SR U R5 U(aU SR U R5 U(dMU SR [U R 55 M X4$)ar Convert the encoding representation (from low-level HuggingFace tokenizer output) to a python Dict and a list of encodings, take care of building a batch from overflowing tokens. Overflowing tokens are converted to additional examples (like batches) so the output values of the dict are lists (overflows) of lists (tokens). Output shape: (overflows, sequence length) token_type_idsattention_mask input_idsspecial_tokens_maskoffset_mappingr) model_input_names overflowingrrUrXidstype_idsrkrmoffsetsr) rr.rbrcrdrerfrgrh encodings encoding_dictrs r]_convert_encoding#TokenizersBackend._convert_encodings( ! ($48N8N$N ! ($48N8N$N ! $)=)=)I! X%9%99I! I#D) A + & - -aee 4$./66qzzB$./66q7G7GH)34;;A This encodes a dummy input and checks the number of added tokens, and is therefore not efficient. Do not put this inside your training loop. Args: pair (`bool`, *optional*, defaults to `False`): Whether the number of added tokens should be computed in the case of a sequence pair or a single sequence. Returns: `int`: Number of special tokens added to sequences. )rnum_special_tokens_to_add)rrs r]r+TokenizersBackend.num_special_tokens_to_adds&88>>r_rqskip_special_tokensc@[U[5(aURRU5$/nU(a[ UR 5O [ 5nUH?n[U5nXT;aMUR URRU55 MA U$)a Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and added tokens. Args: ids (`int` or `list[int]`): The token id (or token ids) to convert to tokens. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Returns: `str` or `list[str]`: The decoded token(s). )rTrrrrall_special_idsrX)rrqrr ids_to_skiprs r]convert_ids_to_tokens'TokenizersBackend.convert_ids_to_tokenss c3  ??..s3 33Fc$../CE EJE# MM$//55e< =   r_textrc HUR"SXUS.UD6R5$)N)r text_pairrrf) _encode_plusr)rrrrrs r]tokenizeTokenizersBackend.tokenize s(  ldOaleklssuur_padding_strategyrrrrrcURRnURRnU[R:XaUbURR 5 OdUUUR URS.n UcSn O"U V s0sHoURU S5_M n n X:waURR"S0U D6 U[R:XaUbURR5 ggU[R:XaUOSn U UbUO URURUR UR"US.n X:waURR$"S0U D6 ggs sn f)a Define the truncation and the padding strategies for fast tokenizers (provided by HuggingFace tokenizers library) and restore the tokenizer settings afterwards. The provided tokenizer has no padding / truncation strategy before the managed section. If your tokenizer set a padding / truncation strategy before, then it will be reset to no padding / truncation when exiting the managed section. Args: padding_strategy ([`~utils.PaddingStrategy`]): The kind of padding that will be applied to the input truncation_strategy ([`~tokenization_utils_base.TruncationStrategy`]): The kind of truncation that will be applied to the input max_length (`int`): The maximum size of a sequence. stride (`int`): The stride to use when handling overflow. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. N)rrrr)rrpad_idr`rrrf)rrvrurDO_NOT_TRUNCATErr0rrrrr DO_NOT_PAD no_padding MAX_LENGTHr pad_token_idr`rr) rrrrrrrrrtargetcurrentrKrs r]set_truncation_and_padding,TokenizersBackend.set_truncation_and_paddingsRBoo00 ??** "4"D"D D&--/) /55!11 F"@FG1kooa66G 11;F; 99 9#**,$$47Q7Q#QZW[F -9-E\4K\K\++!^^#55&8 F!..88"%HsE'rris_split_into_wordsreturn_tensorsrc vSnU"U5(d [S5eUbU"U5(d [S5eU(a@[U[[45=(a" U=(a [US[[45nO[U[[45nU(ay[U[5(a [ S5eUb<[ U5[ U5:wa$[S[ U5S[ U5S35eUb[[X55OUnOU(aX4/OU/n[U[[45(d[ S[U5S 35eURUUUUU U S 9 Uc URnURRU:waUURl URRUUUS 9nUVs/sHnURUU U UUUUUS 9PM nn0nUSSH.nUVVVs/sHunnUUHnUPM M n nnnU UU'M0 UVVVs/sHunnUHnUPM M n!nnnU(a4/n"[U5Hun#un$nU"U#/[ U$S 5-- n"M U"US'US Hn%UR!U%UU5 M [#UU!U S9n&U(dqU cnU(dg[#U&R%5VV's0sH5unn'U[ U'5S:a[U'S[5(aU'SOU'_M7 sn'nU&R&5n&U&$s snfs snnnfs snnnfs sn'nf)Nc[U[5(ag[U[[45(a[ U5S:Xag[US[5(ag[US[[45(a[ US5S:Xd[USS[5(ag[USS[[45(a4[ USS5S:H=(d [USSS[5$ggg)NTrF)rTrYrUrVr)rs r]_is_valid_text_input._is_valid_text_inputqs!S!!Ae}--q6Q;!c**!tUm441Q4yA~AaDGS)A)A##AaDGdE];;"1Q47|q0OJqtAwqz34OO$ r_ztext input must be of type `str` (single example), `list[str]` (batch or single pretokenized example) or `list[list[str]]` (batch of pretokenized examples) or `list[tuple[list[str], list[str]]]` (batch of pretokenized sequence pairs).rzdwhen tokenizing batches of text, `text_pair` must be a list or tuple with the same length as `text`.zbatch length of `text`: z- does not match batch length of `text_pair`: .z:batch_text_or_text_pairs has to be a list or a tuple (got ))rrrrrr)ris_pretokenized)r.rbrcrdrerfrgrhrloverflow_to_sample_mapping) tensor_type)rrTrUrVrY TypeErrorrzipr0rrrr encode_batchrvr{&_eventual_warn_about_too_long_sequencerrrt)(rrrrrrrrrrrrrbrcrdrerfrgrhrrr is_batchedbatch_text_or_text_pairsrtr.tokens_and_encodingssanitized_tokensrrrrstacksanitized_encodingsrrtoksrlbatched_outputr0s( r]rTokenizersBackend._encode_plusYs0 ($D))W   )=i)H)HW  #D4-8hThjQUVWQX[_afZgFhJ#D4-8J )S))$Tc)n)D .s4yk:I'q*FOEZtC,@'A`d $?H(9':dV $2UDMBBLTRjMkLllmn  ''- 3!1% (  '#'#<#< ?? 0 04H H4HDOO 1OO00 $1/1 $& &  " "!&;&;*C+E'=+ # &  '*1-C&:N&:74DIqQIQ&:EN$) S !.1ES0DWQdqdq0DS %)+ & )*> ? 9D!*qcC[8I4J.JJ*!@=W 9 :)+6I  7 7 :w W7''79LZhin4=V*'5&:&:&<&< Uc%j1nE!Hd9S9S%(Y^^&<(( NW "OS"s L"7L' &L.[SUSS55(a![RR5US'[ USSnU"S(X-S.UD6nU R#XUS9 U Gb@[R"U R55nSU ;aU SHnU SUSnUb"UVs/sHnUR%UU5PM nnUU SUS'UH"nU R'U5nUbM[ S 5e UVs/sHnU R'U5PM snU SUS!'M S"HGnUU ;dM U UunnUb UU;aUUnU R'U5nUc [ S 5eUU/U U'MI U US'[ R"[R"U55n UR(R+5n[,R.Hn[1UU5cM[1UU5nUb UU;aUUnUR2R%US5n[5U[5(a;[UUR6UR8UR:UR<S#S$9UU'MUUU'M UR>(aUR>R+5O/nUbURU5 [AU5S :aUUS%'XS&'URB"S(0UD6$s snfs snf![DaGnS'[GU5;a2UR S&S5 URB"S(0UD6nU UlUsSnA$eSnAff=f))u Trains a tokenizer on a new corpus with the same defaults (in terms of special tokens or tokenization pipeline) as the current one. Args: text_iterator (generator of `list[str]`): The training corpus. Should be a generator of batches of texts, for instance a list of lists of texts if you have everything in memory. vocab_size (`int`): The size of the vocabulary you want for your tokenizer. length (`int`, *optional*): The total number of sequences in the iterator. This is used to provide meaningful progress tracking new_special_tokens (list of `str` or `AddedToken`, *optional*): A list of new special tokens to add to the tokenizer you are training. special_tokens_map (`dict[str, str]`, *optional*): If you want to rename some of the special tokens this tokenizer uses, pass along a mapping old special token name to new special token name in this argument. kwargs (`dict[str, Any]`, *optional*): Additional keyword arguments passed along to the trainer from the 🤗 Tokenizers library. Returns: [`PreTrainedTokenizerFast`]: A new tokenizer of the same type as the original one, trained on `text_iterator`. r3r4Nr/r0r r1r2rrrg)r"r#z;This method does not support this type of tokenizer (found z-) only BPE, Unigram, WordLevel and WordPiece.rMridrHcontinuing_subword_prefixend_of_word_suffixr ByteLevelr: pretokenizersc32# UH nUSS:Hv M g7f)r0rNrf).0 pretokenizers r] .s!(X !(K7(Xsinitial_alphabet)rr)rtrainerrrzQAttempted to set a token in the post processor that does not exist in the mappingrq)rsepT) single_wordlstriprstripr4rrOr+z7multiple values for keyword argument 'tokenizer_object'rf)$rploadsrto_strrhrrmrsrtrXrrWanypre_tokenizers_fastralphabetMODEL_TO_TRAINER_MAPPINGtrain_from_iteratorrrryrrrSPECIAL_TOKENS_ATTRIBUTESr|rrTrrrr4rOrrrrY)r text_iteratorrrnew_special_tokensspecial_tokens_maprrr3r4rMrrr added_tokenrr trainer_classrtrained_tokenizer_jsonrrrr special_tokenspecial_token_fullrOr new_tokenizers r]train_new_from_iterator)TokenizersBackend.train_new_from_iterator6sDDOO$:$:$<=%)).9 '++,<= ' "6 *e 3/1N7 #G ,02N7 #H - G $V , 9g&x0<'0:*73G4D3Ew'0 G $V ,0J J/1N7 #G ,Mn]dNeflNmMno>>   *~g66w' 48JJ3EU\F]^iFj3kN7 #K 0!**4::n+EF 'K!ooi6Gd+Ag&v.);G!-+i2HL^2^); 1)*:;C+,<=cB8LF)5TZ![TZ5"4"8"8"FTZ![FLN#34S9(C!'#,#8#8#?#+", s#"(ouCuntejIDYDYZ_D`ntCuN#34S9%@<"0  N2-m>+F+ +{"\Dvj HCPQFR -t4 $ 8 8 +4 ($$ s04T' T,T11 V;;U=6V<U==Vc v^^SSKmSSKJn SSKJn SSKJn Jm U "SS9S[S [4UU4S jj5nU(d[5(aS nUGbRU(dU(GdCU"U5(Ga5U "US UUUS S US9nS nUb[USS9n[R"U5nSSS5 WRS5nURS5nU(a7U RU5U RS5:aU(a UbUS;aU$O-U(a&U RU5U RS5:aU$S nU(dU(Gd\U"U5(GaNU(aSU;a[!USUS5 U c:[#USS 5(d([!USS 5 [$R'SUS35 U$U S Ld[#USS 5(a[!USS 5 SSKnUR*R-UR/S5SS9nUR0n[3UUR*R45(aUUR0S'U$[3UUR*R65(aUR*R9S S S9nUR*R5UU/5UlU$!,(df  GN =f)a6 Patches mistral related tokenizers with incorrect regex if detected 1) Local file with an associated config saved next to it >> Model type one of the mistral models (on older versions) 2) Remote models on the hub from official mistral models >> Tags including `base_model:.*mistralai` rN) lru_cache)version)rhf_api)maxsizemodel_idrQc>T"5RU5nURb1TRSSR UR55(agg![a gf=f)NFzbase_model:.*mistralairT) model_infortagssearchr)rr/rres r]is_base_mistral?TokenizersBackend._patch_mistral_regex..is_base_mistrals_ ++H5zz%995rwwuzz7JKK   sA A%$A%Tz config.jsonF) cache_dirrlocal_files_only%_raise_exceptions_for_missing_entries'_raise_exceptions_for_connection_errors _commit_hashr,r-transformers_versionrz5.0.0)mistralmistral3voxtral ministralpixtralrz$The tokenizer you are loading from 'a' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.z[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+|[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]+[\p{Ll}\p{Lm}\p{Lo}\p{M}]*|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n/]*|\s*[\r\n]+|\s+(?!\S)|\s+isolated)patternbehavior)r use_regex)r functoolsr packagingrtransformers.utils.hubrrrYboolrrorprqrrparsesetattrr|rrrpre_tokenizersSplitRegexrrTr: Metaspacer)rrpretrained_model_name_or_pathrrrris_localrrrrrrr _config_filemistral_config_detectedf_configrtransformers_model_typersplit_pretokenizercurrent_pretokenizerrrs @@r]r&TokenizersBackend._patch_mistral_regexs* '%> 3  c d    00H ( 4 X/:W*X*X&-#!16;8=) L', #',9Q"iilG:'.{{3I'J$*1++l*C' (GMM:N,ORYR_R_`gRh,h 3?3  )()gmm?\>]^eeH?'$.')EXZ_2`2`I':DA%)3)B)B)H)H * 0 0s!", *I*& ,5+B+B(!"6 8Q8Q8Z8Z[[5G //2"&&:Jr1rrrG_added_tokens_encoder_added_tokens_decoderrTrWrZrmr] DecoderFastr` EncodingFastrrUrvr{r~rrrrrrrrrrrrrrrriPathLikerrr__static_attributes__ __classcell__)rs@r]r'r'Ss * EJ``Da)F    !c !C$J !Z_`cZd ! 46666%%%%)@GCGG A4S>A tCH~  nd38nnn:d3 ?&;::10nc3hn$ FF = '''.2-1*/+0',#-(-( $d{-( $d{ -( $( -( %) -(!%-(-(-( tCH~tL11 2-(^ 7#7#*76d3+;&<6WZ6 ?d?s?*tCyt`cfjknfo`o4vSvd vtvjnorjsvI9)I90I9 I9  I9  $J I9DjI9\gk#',;,F,F2D2T2T!%$))-#'&*-1-1*/+0',#,0)X++d9o=EV@WWX004 ?BTJ[E\\_ccX! X * X 0 X$JXX"X $JXDjXt X $d{X $d{X$(X %)!X"!%#X$%X&'X(#Tk)X, -Xt tCy S %*48 )c?)")'+Tk ) )^&*&* bkk)#s(Od{  t  sCx * CJ BBr_r')Cr$rrpri collectionsrcollections.abcrshutilrtypingrtokenizers.pre_tokenizersrrhuggingface_hubrrrr r r+r rmtokenizers.decodersr r*tokenizers.modelsr rtokenizers.trainersrrrrrrrrintegrations.ggmlrmodeling_gguf_pytorch_utilsrtokenization_utils_baserrrrrrrutilsrr r! get_loggerrzrrSPECIAL_TOKENS_MAP_FILETOKENIZER_CONFIG_FILETIKTOKEN_VOCAB_FILEADDED_TOKENS_FILErrr'PreTrainedTokenizerFastrfr_r]rBs  #$7+-/16*^^.05=@?   H %"3/'( !! (6EXY,-j/j.j\),r_