Ë çÐfãó^—ddlmZmZdZdZdZdZdZd„Zd„Z d „Z d „Z d „Z d „Z ee e e e e d œZy)é)Údata_gym_to_mergeable_bpe_ranksÚload_tiktoken_bpez <|endoftext|>z<|fim_prefix|>z<|fim_middle|>z<|fim_suffix|>z<|endofprompt|>có<—tdddd¬«}ddd|td id œS) NzIhttps://openaipublic.blob.core.windows.net/gpt-2/encodings/main/vocab.bpezLhttps://openaipublic.blob.core.windows.net/gpt-2/encodings/main/encoder.jsonÚ@1ce1664773c50f3e0cc8842619a93edc4624525b728b188a9e0be33b7726adc5Ú@196139668be63f3b5d6574427317ae82f612a97c5d1cdaf36ed2256dbf636783)Úvocab_bpe_fileÚencoder_json_fileÚvocab_bpe_hashÚencoder_json_hashÚgpt2éQÄúG'(?:[sdmt]|ll|ve|re)| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+éPÄ©ÚnameÚexplicit_n_vocabÚpat_strÚmergeable_ranksÚspecial_tokens)rÚ ENDOFTEXT©rs úY/opt/lhia/ganansol/python/venv/lib/python3.12/site-packages/tiktoken_ext/openai_public.pyr r s;€Ü5ØbØhØYØ\ô €OðØ!ðbØ*Ü$ eÐ,ñ ð ócó8—tdd¬«}ddd|tdidœS) NzGhttps://openaipublic.blob.core.windows.net/encodings/r50k_base.tiktokenÚ@306cd27f03c1a714eca7108e03d66b7dc042abe8c258b44c199a7ed9838dd930©Ú expected_hashÚ r50k_baser rrr©rrrs rrró2€Ü'ØQØXô€Oð Ø!ØaØ*Ü$ eÐ,ñ  ðrcó8—tdd¬«}ddd|tdidœS) NúGhttps://openaipublic.blob.core.windows.net/encodings/p50k_base.tiktokenÚ@94b5ca7dff4d00767bc256fdd1b27e5b17361d7b8a5f968547f9f23eb70d2069rÚ p50k_baseéiÄrrrrrs rr$r$+r rcó^—tdd¬«}tdtdtdtdi}dd ||d œS) Nr"r#rrr%ijÄikÄÚ p50k_editr©rrrr)rrÚ FIM_PREFIXÚ FIM_MIDDLEÚ FIM_SUFFIX©rrs rr'r'9sA€Ü'ØQØXô€Oô  ¬ °E¼:ÀuÌjÐZ_Ð`€NàØaØ*Ø(ñ  ðrc ój—tdd¬«}tdtdtdtdt di}d d ||d œS) NzIhttps://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktokenÚ@223921b76ee99bde995b7ff738513eef100fb51d18c93597a113bcffe865b2a7ri¡‡i¢‡i£‡i¤‡i´‡Ú cl100k_basezm'(?i:[sdmt]|ll|ve|re)|[^\r\n\p{L}\p{N}]?+\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]++[\r\n]*|\s*[\r\n]|\s+(?!\S)|\s+r()rrr)r*r+Ú ENDOFPROMPTr,s rr/r/GsO€Ü'ØSØXô€Oô 6ÜFÜFÜFÜVð €NððHØ*Ø(ñ  ðrcól—tdd¬«}tdtdi}djgd¢«}d|||d œS) NzHhttps://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktokenÚ@446a9538cb6c348e3516120d7c08b09f57c36495e2acfffe59a5bf8b0cfb1a2dri? iR ú|)zi[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+(?i:'s|'t|'re|'ve|'m|'ll|'d)?zi[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]+[\p{Ll}\p{Lm}\p{Lo}\p{M}]*(?i:'s|'t|'re|'ve|'m|'ll|'d)?z \p{N}{1,3}z ?[^\s\p{L}\p{N}]+[\r\n/]*z \s*[\r\n]+z \s+(?!\S)z\s+Ú o200k_baser()rrr0Újoin)rrrs rr4r4[sR€Ü'ØRØXô€Oô 6ÜVð€Nð h‰hò ó €GðØØ*Ø(ñ  ðr)r rr$r'r/r4N)Ú tiktoken.loadrrrr)r*r+r0r rr$r'r/r4ÚENCODING_CONSTRUCTORS©rrúr9s[ðßLà € Ø € Ø € Ø € Ø€ òò& ò ò òò(ð< ØØØØØñ Ñr