h1JddlmZddlZddlZddlmZmZddlmZm Z m Z ddlm Z ddl mZddlmZddlmZmZmZmZmZmZdd lmZmZej6eZed d ZGd d eeZGddeZ Gdde!eZ"edGddZ#ddZ$y)) annotationsN)ABCabstractmethod) CollectionIterableSequence)Set) dataclass)Enum)AnyCallableLiteralOptionalTypeVarUnion)BaseDocumentTransformerDocumentTS TextSplitter)boundceZdZdZddedddf ddZeddZ d dd Zdd Z dd Z dd Z e dd Z e ddedf ddZ ddZy)rz)Interface for splitting text into chunks.iFTc|dkrd|}t||dkrd|}t|||kDrd|d|d}t|||_||_||_||_||_||_y)adCreate a new TextSplitter. Args: chunk_size: Maximum size of chunks to return chunk_overlap: Overlap in characters between chunks length_function: Function that measures the length of given chunks keep_separator: Whether to keep the separator and where to place it in each corresponding chunk (True='start') add_start_index: If `True`, includes chunk's start index in metadata strip_whitespace: If `True`, strips whitespace from the start and end of every document rzchunk_size must be > 0, got z chunk_overlap must be >= 0, got zGot a larger chunk overlap (z) than chunk size (z), should be smaller.N) ValueError _chunk_size_chunk_overlap_length_function_keep_separator_add_start_index_strip_whitespace)self chunk_size chunk_overlaplength_functionkeep_separatoradd_start_indexstrip_whitespacemsgs \/opt/lhia/marcimex/python/venv/lib/python3.12/site-packages/langchain_text_splitters/base.py__init__zTextSplitter.__init__s* ?0 =CS/ ! 1 4]ODCS/ ! : %.}o><46 S/ !%+ /- /!1cy)z$Split text into multiple components.N)r!texts r) split_textzTextSplitter.split_textEsr+Nc |xsigt|z}g}t|D]\}}d}d}|j|D]} tj||} |j r>||z|j z } |j| td| }|| d<t| }t| | } |j| |S)z&Create documents from a list of texts.r start_index) page_contentmetadata) len enumerater/copydeepcopyrrfindmaxrappend) r!texts metadatas _metadatas documentsir.indexprevious_chunk_lenchunkr3offsetnew_docs r)create_documentszTextSplitter.create_documentsIs32$U"3   ' *GAtE!" . *==A7(("%77$:M:MMF IIeSF^r;r<docs r)split_documentszTextSplitter.split_documents]sVry +C LL)) *   S\\ * +$$Ui$@@r+cl|j|}|jr|j}|dk(ry|S)N)joinr strip)r!docs separatorr.s r) _join_docszTextSplitter._join_docses3~~d#  ! !::>B>N>N=OQ{#a'//+yAC C( $"5"55 [9IA9M STU**+!AI!6!6{1~!F-0-=-AMq"'2!"o  $"5"55 [9IA9M STU**+!AI   q ! Tc+.>.B]J JE9 K:ook95 ? KK  r+c  ddlm}t|s d}t|dfd }|dd|i|S#t$rd}t|wxYw) z>Text splitter that uses HuggingFace tokenizer to count length.r)PreTrainedTokenizerBasezATokenizer received was not an instance of PreTrainedTokenizerBasec8tj|SN)r4tokenizer. tokenizers r)_huggingface_tokenizer_lengthzNTextSplitter.from_huggingface_tokenizer.._huggingface_tokenizer_lengths9--d344r+z`Could not import transformers python package. Please install it with `pip install transformers`.r$r.strreturnintr-)$transformers.tokenization_utils_baser\ isinstancer ImportError)clsrakwargsr\r(rbs ` r)from_huggingface_tokenizerz'TextSplitter.from_huggingface_tokenizersm " Ti)@AW!o% 5K#@KFKK  "E S/ !  "s %2A gpt2allc  ddl}||j| n|j| d fd }t |t r ||d} i|| }|dd|i|S#t$rd}t|wxYw) z9Text splitter that uses tiktoken encoder to count length.rNzCould not import tiktoken python package. This is needed in order to calculate max_tokens_for_prompt. Please install it with `pip install tiktoken`.c>tj|SN)allowed_specialdisallowed_special)r4encode)r.rrrsencs r)_tiktoken_encoderz=TextSplitter.from_tiktoken_encoder.._tiktoken_encoders* $3'9 r+) encoding_name model_namerrrsr$rcr-)tiktokenriencoding_for_model get_encoding issubclassTokenTextSplitter) rjrwrxrrrsrkryr(rv extra_kwargsrus `` @r)from_tiktoken_encoderz"TextSplitter.from_tiktoken_encoders #   !--j9C'' 6C  c, -!.(#2&8 L 0/,/F?#4???? #A  c" "  #s AA4c 6|jt|S)z2Transform sequence of documents by splitting them.)rHlist)r!r>rks r)transform_documentsz TextSplitter.transform_documentss##DO44r+)r"rfr#rfr$zCallable[[str], int]r%z$Union[bool, Literal['start', 'end']]r&boolr'rreNoner.rdre list[str]r^)r;rr<zOptional[list[dict[Any, Any]]]relist[Document])r>zIterable[Document]rer)rMrrNrdre Optional[str])rTz Iterable[str]rNrdrer)rar rkr rer)rjztype[TS]rwrdrxrrr'Union[Literal['all'], AbstractSet[str]]rs&Union[Literal['all'], Collection[str]]rkr rer)r>Sequence[Document]rkr rer)__name__ __module__ __qualname____doc__r4r*rr/rErHrOrZ classmethodrlsetrrr-r+r)rrsQ3 03?D %!%&2&2&2. &2 = &2  &2&2 &2P33MQ+I (A(TLL,$$(CF5EJ *@ *@*@"*@A *@ C *@  *@ *@*@X5+57:5 5r+cVeZdZdZddedf dfd ZddZxZS) r}z/Splitting text to tokens using model tokenizer.rmNrnc t |di| ddl}||j |}n|j |}||_||_||_y#t$rd}t|wxYw)zCreate a new TextSplitter.rNzCould not import tiktoken python package. This is needed in order to for TokenTextSplitter. Please install it with `pip install tiktoken`.r-) superr*ryrirzr{ _tokenizer_allowed_special_disallowed_special) r!rwrxrrrsrkryr(ru __class__s r)r*zTokenTextSplitter.__init__s "6" #   !--j9C'' 6C /#5  #A  c" "  #s AA(cdfd }tjjjj|}t ||S)aSplits the input text into smaller chunks based on tokenization. This method uses a custom tokenizer configuration to encode the input text into tokens, processes the tokens in chunks of a specified size with overlap, and decodes them back into text chunks. The splitting is performed using the `split_text_on_tokens` function. Args: text (str): The input text to be split into smaller chunks. Returns: List[str]: A list of text chunks, where each chunk is derived from a portion of the input text based on the tokenization and chunking rules. chjj|jjSrq)rrtrr)_textr!s r)_encodez-TokenTextSplitter.split_text.._encodes4??)) $ 5 5#'#;#;* r+)r#tokens_per_chunkdecodertr`)rrdrez list[int]) Tokenizerrrrrsplit_text_on_tokens)r!r.rras` r)r/zTokenTextSplitter.split_textsE  --!--??))  $CCr+) rwrdrxrrrrrsrrkr rerr)rrrrrr*r/ __classcell__)rs@r)r}r}s]9$$(CF5EJ 66"6A 6 C 6  6 68Dr+r}c|eZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd ZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZy)Languagez"Enum of the programming languages.cppgojavakotlinjstsphpprotopythonrstrubyrustscalaswiftmarkdownlatexhtmlsolcsharpcobolcluaperlhaskellelixir powershell visualbasic6N)rrrrCPPGOJAVAKOTLINJSrPHPPROTOPYTHONRSTRUBYRUSTSCALASWIFTMARKDOWNLATEXHTMLSOLCSHARPCOBOLCLUAPERLHASKELLELIXIR POWERSHELL VISUALBASIC6r-r+r)rr"s, C B D F B B C E F C D D E EH E D C F E A C DG FJ!Lr+rT)frozenc@eZdZUdZded< ded< ded< ded<y ) rzTokenizer data class.rfr#rzCallable[[list[int]], str]rzCallable[[str], list[int]]rtN)rrrr__annotations__r-r+r)rrBs)*, &&= &&=r+rcg}|j|}d}t||jzt|}|||}|t|kr|j |j ||t|k(r |S||j|j z z }t||jzt|}|||}|t|kr|S)z6Split incoming text and return chunks using tokenizer.r)rtminrr4r:rr#)r.rarT input_ids start_idxcur_idx chunk_idss r)rrPsF  &II)i888#i.IG)G,I c)n $ i&&y12 c)n $  M Y//)2I2III i)"<"<rs" #::.!G   8 $ T(E5*CE5P=D =D@"sD"@ $ > > >r+