Q iBddlmZmZddlmZddlmZddlmZm Z m Z m Z m Z m Z ddlZddlmZmZmZddlmZddlmZmZdd lmZdd lmZdd lmZeeZGd d eZ GddeZ!GddeZ"dS))MappingSequence)Enum)Path)AnyDictListLiteralOptionalUnionN) BaseModelFieldmodel_validator)r) BaseField FieldFactory) TypeInferrer) get_logger model_to_dictceZdZdZdZdZdS) StorageTypez Enumeration for the storage types supported in Redis. Attributes: HASH (str): Represents the 'hash' storage type in Redis. JSON (str): Represents the 'json' storage type in Redis. hashjsonN)__name__ __module__ __qualname____doc__HASHJSONC:\Users\Dell Inspiron 16\Desktop\tws\AgrotaPowerBi\back-agrota-powerbi\mcp-client-agrota\venv\Lib\site-packages\redisvl/schema/schema.pyrrs$ D DDDr!rceZdZUdZeed< dZeeeefed< dZ eed< e j Z e ed< dZ eeeed <dS) IndexInfoaIndex info includes the essential details regarding index settings, such as its name, prefix, key separator, storage type, and stopwords in Redis. In yaml format, the index info section looks like: .. code-block:: yaml index: name: user-index prefix: user key_separator: ':' storage_type: json stopwords: [] # Disable stopwords (STOPWORDS 0) In dict format, the index info section looks like: .. code-block:: python {"index": { "name": "user-index", "prefix": "user", "key_separator": ":", "storage_type": "json", "stopwords": ["the", "a", "an"] # Custom stopwords }} namervlprefix: key_separator storage_typeN stopwords)rrrrstr__annotations__r'r r r)rrr*r+r r r!r"r$r$s8 III'$)FE#tCy. !))) M3? + 0L+000B%)IxS ")))^^r!r$c heZdZUdZeed< eeZe e e fed< dZ e ded< ede fdZed ed e e efde e effd Zed e ddfdZede e efddfdZedee fdZedeefdZde e effdZdee e effdZde fdZdgfde e efdedee dee e effdZde e effdZ d!d e deddfd Z!dS)" IndexSchemaaJA schema definition for a search index in Redis, used in RedisVL for configuring index settings and organizing vector and metadata fields. The class offers methods to create an index schema from a YAML file or a Python dictionary, supporting flexible schema definitions and easy integration into various workflows. An example `schema.yaml` file might look like this: .. code-block:: yaml version: '0.1.0' index: name: user-index prefix: user key_separator: ":" storage_type: json fields: - name: user type: tag - name: credit_score type: tag - name: embedding type: vector attrs: algorithm: flat dims: 3 distance_metric: cosine datatype: float32 Loading the schema for RedisVL from yaml is as simple as: .. code-block:: python from redisvl.schema import IndexSchema schema = IndexSchema.from_yaml("schema.yaml") Loading the schema for RedisVL from dict is as simple as: .. code-block:: python from redisvl.schema import IndexSchema schema = IndexSchema.from_dict({ "index": { "name": "user-index", "prefix": "user", "key_separator": ":", "storage_type": "json", }, "fields": [ {"name": "user", "type": "tag"}, {"name": "credit_score", "type": "tag"}, { "name": "embedding", "type": "vector", "attrs": { "algorithm": "flat", "dims": 3, "distance_metric": "cosine", "datatype": "float32" } } ] }) Note: The `fields` attribute in the schema must contain unique field names to ensure correct and unambiguous field references. index)default_factoryfieldsz0.1.0versionreturnc tjdi|}|tjkr|jr|jn d|j|_n1|j#t d|jdd|_|S)z Parse raw field inputs derived from YAML or dict. Validates and sets the 'path' attribute for fields when using JSON storage type. z$.NzPath attribute for field 'z(' will be ignored for HASH storage type.r )r create_fieldrrpathr%loggerwarning)r* field_inputsfields r" _make_fieldzIndexSchema._make_fields)99L99 ;+ + +',zH7HEJ7H7HEJJz%eeeeEJ r!before)modevaluesc|di}t|ts td i|}|dg}i}t|tr|D]\}}t|t r,|j|krtd|d|jd|||<Ft|tr?|j |j fi|}|j|krtd|d|jd|||<td|dnt|tr`t|ttfsD|D]@}|j |j fi|}|j|vrtd|jd |||j<Antd t|||d<||d<|S) zV Validate uniqueness of field names and create valid field instances. r0r2zField name mismatch: key 'z' vs field name ''zInvalid field type for 'z': expected BaseField or dictDuplicate field name: z/. Field names must be unique across all fields.z#Fields must be a list or dict, got r )get isinstancer$ritemsrr% ValueErrordictr<r*rr,bytestype) clsr?r0 input_fieldsprepared_fieldsr%r; field_obj field_inputs r"validate_and_create_fieldsz&IndexSchema.validate_and_create_fieldssX 7B''%++ '&&&&Ezz(B// 02 lG , ,! Y+1133   eeY//zT))(]]]PUPZ]]]-2OD))t,,  /0B L Le L LI ~--(aaaPYP^aaa-6OD))$V4VVV! & h / / Y 3,9 9 Y , 4 4 '(:JJkJJ:00$llll/4 ++  4W4 CUCUWWXX X*xw r! file_pathc t|}n%#t$r}td||d}~wwxYw|st d|dt |d5}tj|}| |cdddS#1swxYwYdS)aBCreate an IndexSchema from a YAML file. Args: file_path (str): The path to the YAML file. Returns: IndexSchema: The index schema. .. code-block:: python from redisvl.schema import IndexSchema schema = IndexSchema.from_yaml("schema.yaml") zInvalid file path: N Schema file z does not existr) rresolveOSErrorrFexistsFileNotFoundErroropenyaml safe_loadmodel_validate)rJrPfpef yaml_datas r" from_yamlzIndexSchema.from_yamls Gi((**BB G G G>9>>??Q F Gyy{{ O#$M9$M$M$MNN N "c]] 1aq))I%%i00 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1s&!$ AAA)B66B:=B:datacT|}||S)aCreate an IndexSchema from a dictionary. Args: data (Dict[str, Any]): The index schema data. Returns: IndexSchema: The index schema. .. code-block:: python from redisvl.schema import IndexSchema schema = IndexSchema.from_dict({ "index": { "name": "docs-index", "prefix": "docs", "storage_type": "hash", }, "fields": [ { "name": "doc-id", "type": "tag" }, { "name": "doc-embedding", "type": "vector", "attrs": { "algorithm": "flat", "dims": 1536 } } ] }) )copyr[)rJra schema_dicts r" from_dictzIndexSchema.from_dicts&Hiikk !!+...r!cNt|jS)zA list of field names associated with the index schema. Returns: List[str]: A list of field names from the schema. )listr2keys)selfs r" field_nameszIndexSchema.field_names/s DK$$&&'''r!cLd|jD}|S)arA list of core redis-py field definitions based on the current schema fields. Converts RedisVL field definitions into a format suitable for use with redis-py, facilitating the creation and management of index structures in the Redis database. Returns: List[RedisField]: A list of redis-py field definitions. c<g|]\}}|Sr )as_redis_field).0_r;s r" z,IndexSchema.redis_fields..Ds5* * * '/q%E " "* * * r!)r2rE)ri redis_fieldss r"rqzIndexSchema.redis_fields8s6* * 37;3D3D3F3F* * * r!r:c|j|jjfi|}|j|jvrt d|jd||j|j<dS)aAdds a single field to the index schema based on the specified field type and attributes. This method allows for the addition of individual fields to the schema, providing flexibility in defining the structure of the index. Args: field_inputs (Dict[str, Any]): A field to add. Raises: ValueError: If the field name or type are not provided or if the name already exists within the schema. .. code-block:: python # Add a tag field schema.add_field({"name": "user", "type": "tag"}) # Add a vector field schema.add_field({ "name": "user-embedding", "type": "vector", "attrs": { "dims": 1024, "algorithm": "flat", "datatype": "float32" } }) rBz>. Field names must be unique across all fields for this index.N)r<r0r*r%r2rF)rir:r;s r" add_fieldzIndexSchema.add_fieldIsj>! !8IILII : $ $ssss #( EJr!c:|D]}||dS)a\Extends the schema with additional fields. This method allows dynamically adding new fields to the index schema. It processes a list of field definitions. Args: fields (List[Dict[str, Any]]): A list of fields to add. Raises: ValueError: If a field with the same name already exists in the schema. .. code-block:: python schema.add_fields([ {"name": "user", "type": "tag"}, {"name": "bio", "type": "text"}, { "name": "user-embedding", "type": "vector", "attrs": { "dims": 1024, "algorithm": "flat", "datatype": "float32" } } ]) N)rs)rir2r;s r" add_fieldszIndexSchema.add_fieldsqs2: " "E NN5 ! ! ! ! " "r! field_namech||jvr td|ddS|j|=dS)zRemoves a field from the schema based on the specified name. This method is useful for dynamically altering the schema by removing existing fields. Args: field_name (str): The name of the field to be removed. zField 'z' does not exist in the schemaN)r2r8r9)rirvs r" remove_fieldzIndexSchema.remove_fieldsD T[ ( ( NNOZOOO P P P F K # # #r!Fstrict ignore_fieldsc\g}|D]\}}||vr tj|}|t j||Z#t$r.}|rt d|d|Yd}~d}~wwxYw|S)a,Generates a list of extracted field specs from a sample data point. This method simplifies the process of creating a schema by inferring field types and attributes from sample data. It's particularly useful during the development process while dealing with datasets containing numerous fields, reducing the need for manual specification. Args: data (Dict[str, Any]): Sample data used to infer field definitions. strict (bool, optional): If True, raises an error on failing to infer a field type. Defaults to False. ignore_fields (List[str], optional): A list of field names to exclude from processing. Defaults to an empty list. Returns: Dict[str, List[Dict[str, Any]]]: A dictionary with inferred field types and attributes. Notes: - Vector fields are not generated by this method. - This method employs heuristics and may not always correctly infer field types. zError inferring field type for z: )messageN) rErinferappendrr6 model_dumprFr8warn) riraryrzr2rvvalue field_typer]s r"generate_fieldszIndexSchema.generate_fieldss:(*!%   J]** )/66   -""!jll     KK S* S SPQ S S    sAA11 B);$B$$B)ct|jd|jD|jd}|S)zSerialize the index schema model to a dictionary, handling Enums and other special cases properly. Returns: Dict[str, Any]: The index schema as a dictionary. c2g|]\}}t|Sr r)rnrvr;s r"rpz'IndexSchema.to_dict..s1):U e$$r!)r0r2r3)rr0r2rEr3)ri dict_schemas r"to_dictzIndexSchema.to_dictsV#4:..>Bk>O>O>Q>Q|   r!T overwriteNcBt|}|r|std|dt |d5}|}t j||dddddS#1swxYwYdS)a)Write the index schema to a YAML file. Args: file_path (str): The path to the YAML file. overwrite (bool): Whether to overwrite the file if it already exists. Raises: FileExistsError: If the file already exists and overwrite is False. rRz already exists.wF) sort_keysN)rrTrVFileExistsErrorrXrrYdump)rirPrr\r^r_s r"to_yamlzIndexSchema.to_yamls)__ $ $ & & 99;; Ny N!"L"L"L"LMM M "c]] 5a I Iie 4 4 4 4 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5s,BBB)T)"rrrrr$r-rrGr2rr,rr3r staticmethodr<r classmethodrrOr`repropertyr rj RedisFieldrqrsrurxboolrrrr r!r"r/r/MsIIV 4#(5#>#>#>FDi >>>M!(GWW '''1Y\&_(###2S#X24S>222[$#2h1#1-111[14$/T#s(^$/ $/$/$/[$/L(T#Y(((X(d:.X &(d38n&(&(&(&(P"d38n!5""""@ $s $ $ $ $"#% 0038n00Cy 0 d38n  0000dc3h"5555555555r!r/)#collections.abcrrenumrpathlibrtypingrrr r r r rYpydanticr rrredis.commands.search.fieldrredisvl.schema.fieldsrrredisvl.schema.type_utilsrredisvl.utils.logrredisvl.utils.utilsrrr8rr$r/r r!r"rs--------<<<<<<<<<<<<<<<< 6666666666;;;;;;99999999222222((((((------ H       $   +^+^+^+^+^ +^+^+^\e5e5e5e5e5)e5e5e5e5e5r!