[ǻi> vdZddlmZddlmZddlmZddlmZ ddl Z dZ Gd d eZ y#e $rdZ YwxYw) zNLTK text splitter.) annotations)Any)override) TextSplitterNTFcTeZdZdZ ddd dfdZeddZxZS) NLTKTextSplitterz"Splitting text using NLTK package.F)use_span_tokenizec rt|di|||_||_||_|jr|jr d}t |t s d}t||jr/tjj|j|_ ytjj|_ y)aInitialize the NLTK splitter. Args: separator: The separator to use when combining splits. language: The language to use. use_span_tokenize: Whether to use `span_tokenize` instead of `sent_tokenize`. Raises: ImportError: If NLTK is not installed. ValueError: If `use_span_tokenize` is `True` and separator is not `''`. z6When use_span_tokenize is True, separator should be ''zANLTK is not installed, please install it with `pip install nltk`.N) super__init__ _separator _language_use_span_tokenize ValueError _HAS_NLTK ImportErrornltktokenize_get_punkt_tokenizer _tokenizer sent_tokenize)self separatorlanguager kwargsmsg __class__s [/opt/lhia/marcimex/agent/venv/lib/python3.12/site-packages/langchain_text_splitters/nltk.pyr zNLTKTextSplitter.__init__s( "6"#!"3  " "tJCS/ !UCc" "  " ""mm@@PDO"mm99DOcj|jrot|jj|}g}t |D]:\}\}}|dkDr||dz d}||||||z}n|||}|j |<n|j||j }|j||jS)Nr)r) rlistr span_tokenize enumerateappendr _merge_splitsr) rtextspanssplitsistartendprev_endsentences r split_textzNLTKTextSplitter.split_text9s  " "66t<=EF#,U#3 (r?sC"&6I 5;|5; Is .88