3j04zSSKrSSKrSSKrSSKJr SSKJr SSKJrJ r SSK J r "SS5r "SS \5r g) N)SentencePieceProcessor)PreTrainedTokenizer) BatchEncoding EncodedInput)PaddingStrategyc \rSrSrS\4SjrSS\4SjjrSS\S\S\S\\ 4S jjr S \\ S\4S jr S \\S\4S jr Sr SrSrg) SPTokenizer model_pathc.[RRU5(dU5e[US9UlURR 5UlURR5UlURR5UlURR5Ul URR 5URR5:Xde/SQn/SQU-n0Ul 0Ul UHInUR URU'X@RUR 'U=R S- slMK SRUVs/sHn[R "U5PM sn5Ulgs snf)N) model_file)z <|system|>z<|user|> <|assistant|>z<|observation|>)z[MASK][gMASK]z[sMASK]sopeop|)ospathisfilersp_model vocab_sizen_wordsbos_ideos_idunk_idpad_idget_piece_sizespecial_tokensindex_special_tokensjoinreescaperole_special_token_expression)selfr role_special_tokensrtokens ^/home/wildlama/miniconda3/lib/python3.13/site-packages/diffusers/pipelines/kolors/tokenizer.py__init__SPTokenizer.__init__s6ww~~j))5:5).*E !MM446 ==//1 ==//1 ==//1 }}'')T]]-I-I-KKKK\GJ]] $&!#E)-D   &6; % %dll 3 LLA L$.1XXUh6iUhEryy7GUh6i-j*6is% Fsc 0U(aSn/n[R"URU5HnX5R5:a:UR UR R XUR555 URXR5UR55 UR5nM U[U5:a,UR UR R XS55 U$UR R U5$)Nr) r"finditerr$startextendrEncodeAsPiecesappendendlen)r%r+encode_special_tokens last_indextmatchs r(tokenizeSPTokenizer.tokenize0s JAT%G%GK -HHT]]99!:WXY;;=599;78"YY[ L CF"55a nEFH==//2 2boseosreturnc[U[5(deURRU5nU(aUR/U-nU(aX@R /-nU$N) isinstancestrrencoderr)r%r+r;r<r6s r(rBSPTokenizer.encode?sR!S!!!! MM  #  !A [[M!Ar:r6cS/p2UH\nX@R;a9U(aX RRU5- n/nX RU- nMKURU5 M^ U(aX RRU5- nU$)N)r rdecoder1)r%r6textbufferr's r(rFSPTokenizer.decodeHs~2fE111MM0088DF11%88 e$  MM((0 0D r:tokensc<URRU5nU$r?)r DecodePieces)r%rJrGs r( decode_tokensSPTokenizer.decode_tokensVs}}))&1 r:ctXR;aURU$URRU5$z0Converts a token (str) in an id using the vocab.)rr PieceToIdr%r's r(convert_token_to_idSPTokenizer.convert_token_to_idZs4 '' '&&u- -}}&&u--r:cXR;aURU$XRURUR4;dUS:agURR U5$)=Converts an index (integer) in a token (str) using the vocab.rrE)r rrrr IdToPiecer%indexs r(convert_id_to_tokenSPTokenizer.convert_id_to_token`sV -- -,,U3 3 [[$++t{{; ;uqy}}&&u--r:)rrr rrr$rrN)F)FF)__name__ __module__ __qualname____firstlineno__rAr)r8boollistintrBrFrMrSrZ__static_attributes__r:r(r r syk3k, 3# 3$Td3i S c DI#. .r:r c8^\rSrSrSS0r/SQrS(U4SjjrSr\S\ 4Sj5r \ RS \ 4S j5r \S\ 4S j5r \ RS \ 4S j5r \S 5r \S\ 4Sj5r\RS \ 4Sj5r\S5r\S5rSrSrSrSrS\\ S\ 4SjrS)SjrSrSrS*SjrS)S\\S\\S-S\\4SjjrS\R<SSS4S \\ \ 4\!-S!\S-S"\S#\S-S$\"S-S%\"S-S\4S&jjr#S'r$U=r%$)+ChatGLMTokenizeri vocab_fileztokenizer.model) input_idsattention_mask position_idsc >SUlXl[U5UlURRURR URR S.UlX@l[TU]("SUUUS.UD6 g)N GLMTokenizer)z) padding_sideclean_up_tokenization_spacesr4rd) namerhr tokenizerrrrrr4superr))r%rhrprqr4kwargs __class__s r(r)ChatGLMTokenizer.__init__ns{# $$Z0^^**^^**^^**  &;"  %)E"7   r:cXR;aURU$XRR;dUSUR35eURRU$)Nz is not a special token for )rrsrrrRs r( get_commandChatGLMTokenizer.get_commandsc '' '&&u- -555h%@\]a]f]f\g7hh5~~,,U33r:r=cgNzrdr%s r( unk_tokenChatGLMTokenizer.unk_tokenr:valuecXlgr?) _unk_tokenr%rs r(r~rr:cgr|rdr}s r( pad_tokenChatGLMTokenizer.pad_tokenrr:cXlgr?) _pad_tokenrs r(rrrr:c$URS5$)Nroryr}s r( pad_token_idChatGLMTokenizer.pad_token_id((r:cg)Nzrdr}s r( eos_tokenChatGLMTokenizer.eos_tokensr:cXlgr?) _eos_tokenrs r(rrrr:c$URS5$)Nrnrr}s r( eos_token_idChatGLMTokenizer.eos_token_idrr:c.URR$r?)rsrr}s r(rChatGLMTokenizer.vocab_sizes~~%%%r:c[UR5Vs0sHoRU5U_M nnURUR5 U$s snf)zReturns vocab as a dict)ranger_convert_id_to_tokenupdateadded_tokens_encoder)r%ivocabs r( get_vocabChatGLMTokenizer.get_vocabsL:?:PQ:PQ**1-q0:PQ T../ RsAc HURRXRS9$)N)r4)rsr8r4)r%rGrus r( _tokenizeChatGLMTokenizer._tokenizes ~~&&tC]C]&^^r:c8URRU5$rP)rsrSrRs r(_convert_token_to_id%ChatGLMTokenizer._convert_token_to_id~~11%88r:c8URRU5$)rV)rsrZrXs r(r%ChatGLMTokenizer._convert_id_to_tokenrr:rJc8URRU5$r?)rsrM)r%rJs r(convert_tokens_to_string)ChatGLMTokenizer.convert_tokens_to_strings~~++F33r:Nc[RRU5(a-[RRXRS5nOUn[ UR S5nUR5nSSS5 [ US5nURW5 SSS5 U4$!,(df  N7=f!,(df  U4$=f)a9 Save the vocabulary and special tokens file to a directory. Args: save_directory (`str`): The directory in which to save the vocabulary. filename_prefix (`str`, *optional*): An optional prefix to add to the named of the saved files. Returns: `tuple(str)`: Paths to the files saved. rhrbNwb) rrisdirr!vocab_files_namesopenrhreadwrite)r%save_directoryfilename_prefixrhfin proto_strwriters r(save_vocabulary ChatGLMTokenizer.save_vocabularys 77== ( (n6L6L\6Z[J'J $//4 (C I)*d #v LL #$} ) ($ #}s*B,B=, B:= C cJURS5URS5/nU$)Nrrr)r% prefix_tokenss r(get_prefix_tokens"ChatGLMTokenizer.get_prefix_tokenss()))4d6F6Fu6MN r:cUS;dU5eURSUS35/URRUS35-nURRU5nXE-nU$)N)systemuser assistant observationz<|z|> )ryrsrB)r%rolemetadatamessage role_tokensmessage_tokensrJs r(build_single_message%ChatGLMTokenizer.build_single_messagespEEKtKE''"TF" 67$..:O:OS[R\\^P_:`` ..w7- r:c Uc/n/nUHknUSnUSS:Xa%SU;aUS-[R"USSSS9-nURURUSUR S S 5U55 Mm URURUS U55 URUR S 5/5 UR U/S S S9$)NcontentrrtoolsrF)indent ensure_asciirrErptT)return_tensorsis_split_into_words)jsondumpsr/rgetrybatch_encode_plus)r%queryhistoryrriitemrs r(build_chat_input!ChatGLMTokenizer.build_chat_inputs ?G D9oGF|x'GtO!D.4::d7mA\a+bb   T66tF|TXXjZ\E]_fg h  224UCD$**?;<=%%yk$\`%aar: token_ids_0 token_ids_1cbUR5nX1-nUbX-URS5/-nU$)a6 Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BERT sequence has the following format: - single sequence: `[CLS] X [SEP]` - pair of sequences: `[CLS] A [SEP] B [SEP]` Args: token_ids_0 (`list[int]`): list of IDs to which the special tokens will be added. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `list[int]`: list of [input IDs](../glossary#input-ids) with the appropriate special tokens. rn)rry)r%rrrs r( build_inputs_with_special_tokens1ChatGLMTokenizer.build_inputs_with_special_tokenssA&..0 #1  "%3t7G7G7P6QQKr:encoded_inputs max_lengthpadding_strategypad_to_multiple_ofreturn_attention_maskrpc:URS:XdeXRSn[U5nU[R:Xa [U5nUbUbX$-S:wa X$-S-U-nU[R :g=(a [U5U:gn SU;a S/U-US'SU;a[ [U55US'U (aZU[U5- n SU;aS/U -US-US'SU;aS/U -US-US'UR/U -U-XRS'U$)a Pad encoded inputs (on left/right and up to predefined length or max length in the batch) Args: encoded_inputs: Dictionary of tokenized inputs (`list[int]`) or batch of tokenized inputs (`list[list[int]]`). max_length: maximum length of the returned list and optionally padding length (see below). Will truncate by taking into account the special tokens. padding_strategy: PaddingStrategy to use for padding. - PaddingStrategy.LONGEST Pad to the longest sequence in the batch - PaddingStrategy.MAX_LENGTH: Pad to the max length (default) - PaddingStrategy.DO_NOT_PAD: Do not pad The tokenizer padding sides are defined in self.padding_side: - 'left': pads on the left of the sequences - 'right': pads on the right of the sequences pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability `>= 7.5` (Volta). padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. return_attention_mask: (optional) Set to False to avoid returning attention mask (default: set to model specifics) leftrrrjrk) rpmodel_input_namesr3rLONGEST DO_NOT_PADrarr) r%rrrrrrprequired_input seq_lengthneeds_to_be_padded differences r(_padChatGLMTokenizer._pads[H  F***'(>(>q(AB( 66 6^,J  !&8&D*JimnJn%;q@DVVJ-1K1KKqPSTbPcgqPq > 101sZ/?N+ ,  /-1% 2C-DN> * #c.&99J>14533CnUeFf3f/0/23z1ANSaDb1b~.9=9J9J8Kj8X[i8iN11!4 5r:)rrrr4rrrrsrh)rFFr?)Nr)&r\r]r^r_rrr)rypropertyrAr~setterrrrrrrrrrrarrrrrrbrrrdictrrr`rrc __classcell__)rvs@r(rfrfis%'89G %*#  24 3 s  3 s  ))3 s  ))&& _994tCy4S44 bGK9379t3C c8"&,;,F,F)--1$(AS,./-?A$JA* A  $J A $d{ ATkA AAr:rf)rrr" sentencepiecer transformersr$transformers.tokenization_utils_baserrtransformers.utilsrr rfrdr:r(rs8 0,L.M.M.`h*hr: