Q iejddlmZddlmZmZmZmZmZmZm Z m Z m Z m Z ddl mZmZddlmZddlmZddlmZejdrddlmZnddlmZdd lZdd lmZdd l m!Z!dd l"m#Z#dd l$m%Z%ddl&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,ddl-m.Z.ddl/m0Z0e.e1Z2GddeZ3Gdde3Z4Gdde3Z5d S)) Collection) Any AwaitableCallableDictIterableListOptionalTupleUnioncast) BaseModelValidationError) __version__)Pipeline)ClusterPipeline5) IndexTypeN)SchemaValidationError) convert_bytes) IndexSchema)validate_object)AsyncRedisClientAsyncRedisClientOrPipelineAsyncRedisPipelineRedisClientOrPipelineSyncRedisClientSyncRedisPipeline) get_logger) create_ulidcPeZdZUdZeed< eed< dZeed< e de de de d e fd Z d)d e e e fd ee d e fdZe d)d e deed e e e ffdZe dede d e e e fd eee e e fffdZe dede d e e e fd eee e e fffdZe dede d eee e e fffdZe dede d eee e e fffdZe dede ded d fdZd e e e fd e e e ffdZ d*dee deee d ee d ee fdZde ded e e e fd e fdZ! d+dee d ee deee deed!e"d ee#e e e e fff d"Z$ d,d#e%dee d ee deee deedeed$eed!e"d ee fd%Z& d,d#e'dee d ee deee deed$eedeed!e"d ee fd&Z( d)d#e%de)e d$eed ee e e ffd'Z* d)d#e'de)e d$eed ee e e ffd(Z+d S)- BaseStoragez Base class for internal storage handling in Redis. Provides foundational methods for key management, data preprocessing, validation, and basic read/write operations (both sync and async). type index_schemadefault_batch_sizeidprefix key_separatorreturncT|s|S|r||n|}|r|||S|S)aCreate a Redis key using a combination of a prefix, separator, and the identifider. Args: id (str): The unique identifier for the Redis entry. prefix (str): A prefix to append before the key value. key_separator (str): A separator to insert between prefix and key value. Returns: str: The fully formed Redis key. )rstrip)r'r(r)normalized_prefixs C:\Users\Dell Inspiron 16\Desktop\tws\AgrotaPowerBi\back-agrota-powerbi\mcp-client-agrota\venv\Lib\site-packages\redisvl/index/storage.py_keyzBaseStorage._keyCsX I1>I m,,,6 ! +@]@B@@@ Nobjid_fieldc,|t}n- ||}n##t$rtd|d|wxYw|jjj}t |tr|dn|}||||jjj S)aConstruct a Redis key for a given object, optionally using a specified field from the object as the key. Args: obj (Dict[str, Any]): The object from which to construct the key. id_field (Optional[str], optional): The field to use as the key, if provided. Returns: str: The constructed Redis key for the object. Raises: ValueError: If the id_field is not found in the object. Nz Key field z not found in record r)r(r)) r KeyError ValueErrorr$indexr( isinstancelistr/r))selfr1r2 key_valuer(r-s r. _create_keyzBaseStorage._create_key^s  # II TM  T T T !Rh!R!RS!R!RSSS T"(/)3FD)A)AMF1IIvyy $+1?   s < preprocessc |r ||}|S)a5Apply a preprocessing function to the object if provided. Args: preprocess (Optional[Callable], optional): Function to process the object. obj (Any): Object to preprocess. Returns: Dict[str, Any]: Processed object as a dictionary. )r1r<s r. _preprocesszBaseStorage._preprocesss  "*S//C r0clientkeyct)aSynchronously set the value in Redis for the given key. Args: client (RedisClientOrPipeline): The Redis client instance. key (str): The key under which to store the object. obj (Dict[str, Any]): The object to store in Redis. NotImplementedErrorr@rAr1s r._setzBaseStorage._sets "!r0cKt)zGAsynchronously set data in Redis using the provided client or pipeline.rCrEs r._asetzBaseStorage._aset "!r0ct)zHSynchronously get data from Redis using the provided client or pipeline.rCr@rAs r._getzBaseStorage._gets "!r0cKt)zIAsynchronously get data from Redis using the provided client or pipeline.rCrKs r._agetzBaseStorage._agetrIr0ttlcKt|ttfr|||dS|||d{VdS)a1Asynchronously set TTL on a key using the provided client or pipeline Args: client (AsyncRedisClientOrPipeline): The async Redis client or pipeline instance. key (str): The key for which to set the TTL. ttl (int): Time-to-live in seconds for each key. N)r7 AsyncPipelineAsyncClusterPipelineexpire)r@rArOs r._aexpirezBaseStorage._aexpiresf f}.BC D D * MM#s # # # # #--S)) ) ) ) ) ) ) ) ) )r0c,t|j|S)a Validate an object against the schema using Pydantic-based validation. Args: obj: The object to validate Returns: Validated object with any type coercions applied Raises: ValueError: If validation fails )rr$)r9r1s r. _validatezBaseStorage._validatest0#666r0objectskeysc0g}|rt|nd}|rt| tr d |d | d } nd |d t | jd | d} n|dkrd |d| dt | jd} n|dkrd |dt | jd | d} nk|dkrd |dt | jd | d} nFd|vr d|d|} n'd|vr d|d|} nd |d|} |d| at|d kr|d! d} nd"d#|z} d$|d%| d&tj |td'(d)d*tt|d*krd+nd,d-S).a^ Create a human-readable error message from a Pydantic ValidationError. Args: validation_error: The Pydantic ValidationError obj_index: The index of the object that failed validation obj: The object that failed validation Returns: A detailed, actionable error message .c34K|]}t|VdSN)str).0locs r. zHBaseStorage._create_readable_validation_error_message..s(!C!Cs#c((!C!C!C!C!C!Cr0rir#msginputzN/A bytes_typezField 'z6' expects bytes (vector data), but got boolean value 'z'. If this should be a vector field, provide a list of numbers or bytes. If this should be a different field type, check your schema definition.z'' expects bytes (vector data), but got z value 'z9'. For vector fields, provide a list of numbers or bytes. bool_typez' cannot be boolean. Got 'z ' of type z(. Provide a valid numeric value instead. string_typez' expects a string, but got z:'. Convert the value to a string or check your data types. list_typez,' expects a list (for vector data), but got z+'. Provide the vector as a list of numbers. dimensionszVector field 'z' has incorrect dimensions. rangez(' has values outside the allowed range. z': u • rzMultiple validation errors:  z-Schema validation failed for object at index z. z Object data: )defaultindentNr%z...z{ Hint: Check that your data types match the schema field definitions. Use index.schema.fields to view expected field types.)errorsjoingetr7boolr#__name__lowerr]r[stripjsondumpsrg) r9rarbr1 error_detailserror field_name error_type error_msg input_value suggestion detail_msgs r.)_create_readable_validation_error_messagez5BaseStorage._create_readable_validation_error_messagesg %,,..+ 8+ 8E!C!CeEl!C!C!CCCJvJe I))GU33K\))k400 c*ccdocccJR*RRUYZeUfUfUoRRzERRRJ{**>j>>K>>[_`k[l[l[u>>> },,OjOOd;FWFWF`OOjuOOO {**@j@@VZ[fVgVgVp@@{F@@@ !2!222XZXXYXX IOO----mjmmbkmm AzAAiAA  !6*!6!6 7 7 7 7 }   " "&q)//99JJ8499];S;SSJ EI E E E E JsCBBB4C4H ESVWZ[^W_W_S`S`cfSfSf%%ln E E E r0Fvalidatec g}|rt|nd}t|D]\}} |rt|n|| |} || |} |r|| } || | fr#t$r,} || || } t| | d} ~ wt$r-} d|}td|dt| | d} ~ wwxYw|S)ao Preprocess and validate a list of objects with fail-fast approach. Args: objects: List of objects to preprocess and validate id_field: Field to use as the key keys: Optional iterable of keys preprocess: Optional preprocessing function validate: Whether to validate against schema Returns: List of tuples (key, processed_obj) for valid objects Raises: SchemaValidationError: If validation fails, with context about which object failed ValueError: If any other processing errors occur Nz at index zError processing object z: ) rZ enumerater\r;r?rVr]rrr Exceptionr5rg)r9rWr2rXr<rprepared_objectsr_ir1rA processed_objedetailed_message object_ids r. _preprocess_and_validate_objectsz,BaseStorage._preprocess_and_validate_objects1sg2&*4T  ((  FAs %9D'''))#x88!% 0 0j A A B$(NN=$A$AM!''m(<====" E E E#'#Q#Qq#$$ ,,<==1D   +OO  DyDDCFFDD  s$A+B D!'C D(C==D redis_client batch_sizec z|r/t|t|krtd||j}|sgS|t |||||} g} |d5} t | dD]i\} \} }|| | ||r| | || | | |zdkr| jt| |zdkr| dddn #1swxYwY| S) aWrite a batch of objects to Redis as hash entries. This method returns a list of Redis keys written to the database. Args: redis_client (RedisClient): A Redis client used for writing data. objects (Iterable[Any]): An iterable of objects to store. id_field (Optional[str], optional): Field used as the key for each object. Defaults to None. keys (Optional[Iterable[str]], optional): Optional iterable of keys, must match the length of objects if provided. ttl (Optional[int], optional): Time-to-live in seconds for each key. Defaults to None. preprocess (Optional[Callable], optional): A function to preprocess objects before storage. Defaults to None. batch_size (Optional[int], optional): Number of objects to write in a single Redis pipeline execution. validate (bool, optional): Whether to validate objects against schema. Defaults to False. Raises: ValueError: If the length of provided keys does not match the length of objects, or if validation fails. 3Length of keys does not match the length of objectsNr2rXr<rF transactionrsstartr) r[r5r&rr8pipelinerrFrSr]execute)r9rrWr2rXrOr<rrr added_keyspiperrAr1s r.writezBaseStorage.writeosD  TCIIW--RSS S  0J I @@ MM! A    " "u " 5 5 !*+;1!E!E!E # # :C $S)))*KKS)))!!#&&&z>Q&&LLNNN#$$z1Q66 !               $s>B&D00D47D4c K|r/t|t|krtd||j}|sgS|t |||||} g} |d4d{V} t | dD]|\} \} }|| | |d{V|r|| | |d{V| | | |zdkr| d{V}t| |zdkr| d{Vdddd{Vn#1d{VswxYwY| S) aPAsynchronously write objects to Redis as hash entries using pipeline batching. The method returns a list of keys written to the database. Args: redis_client (AsyncRedisClient): An asynchronous Redis client used for writing data. objects (Iterable[Any]): An iterable of objects to store. id_field (Optional[str], optional): Field used as the key for each object. Defaults to None. keys (Optional[Iterable[str]], optional): Optional iterable of keys. Must match the length of objects if provided. ttl (Optional[int], optional): Time-to-live in seconds for each key. Defaults to None. batch_size (Optional[int], optional): Number of objects to write in a single Redis pipeline execution. preprocess (Optional[Callable], optional): An async function to preprocess objects before storage. Defaults to None. validate (bool, optional): Whether to validate objects against schema. Defaults to False. Returns: List[str]: List of Redis keys loaded to the databases. Raises: ValueError: If the length of provided keys does not match the length of objects, or if validation fails. rNrFrrsrr) r[r5r&rr8rrrHrTr]r)r9rrWr2rXrOrr<rrrrrrAr1s r.awritezBaseStorage.awritesWL  TCIIW--RSS S  0J I @@ MM! A   ((U(;; % % % % % % %t!*+;1!E!E!E ) ) :CjjsC0000000008--c3777777777!!#&&&z>Q&&,,..(((((((#$$z1Q66llnn$$$$$$$! % % % % % % % % % % % % % % % % % % % % % % % % % % %$sB?E E!$E!cg}t|tstdt|dkrgS||j}|d5}t |dD]K\}}|||||zdkr'|| L||zdkr'|| dddn #1swxYwYt|S)aRetrieve objects from Redis by keys. Args: redis_client (SyncRedisClient): Synchronous Redis client. keys (Collection[str]): Keys to retrieve from Redis. batch_size (Optional[int], optional): Number of objects to write in a single Redis pipeline execution. Defaults to class's default batch size. Returns: List[Dict[str, Any]]: List of objects pulled from redis. $Keys must be a collection of stringsrNFrrsr) r7r TypeErrorr[r&rrrLextendrrr9rrXrresultsrrrAs r.r{zBaseStorage.get sO$$ ++ DBCC C t99>>I  0J " "u " 5 5 /#D222 3 33 $$$$z>Q&&NN4<<>>222:~""t||~~...  / / / / / / / / / / / / / / /W%%%sBC55C9<C9cjKg}t|tstdt|dkrgS||j}|d4d{V}t |dD]W\}}|||d{V||zdkr-|| d{VX||zdkr-|| d{Vdddd{Vn#1d{VswxYwYt|S)aAsynchronously retrieve objects from Redis by keys. Args: redis_client (AsyncRedisClient): Asynchronous Redis client. keys (Collection[str]): Keys to retrieve from Redis. batch_size (Optional[int], optional): Number of objects to write in a single Redis pipeline execution. Defaults to class's default batch size. Returns: Dict[str, Any]: Dictionary with keys and their corresponding objects. rrNFrrsr) r7rrr[r&rrrNrrrrs r.agetzBaseStorage.aget2s&$ ++ DBCC C t99>>I  0J ((U(;; 5 5 5 5 5 5 5t#D222 9 93jjs+++++++++z>Q&&NN#7#7#7#7#7#7888:~""T\\^^333333444  5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5W%%%s#B D D"Drf)NN)NNNF)NNNNNF),r} __module__ __qualname____doc__r__annotations__rr&int staticmethodrgr/rrr r;rr?rr rrFrrrHrLrNrTrVr rr`rrr|r rrrrrrr{rr>r0r.r"r"4s OOO'!!!!!+c##\4  tCH~ # RU    B(8*<SRUX\  "% ",/ "6:38n "  $sCx.0 1 " " "\ ""*"14";?S>" !4S>1 2"""\" "%",/"  $sCx.0 1"""\" "*"14" !4S>1 2"""\"  *9 * *# *RV * * *\ *7T#s(^7S#X7777&)-"& cx}%3-  c .H /H >#&& &,,,,,,,, ,r0N)r}rrrrHASHr#rrrrgrrrFrrHrrLr rrNr>r0r.rr]s: nD)$$$&+*++4S>+++\+ 56 5S 5tCQTH~ 5 5 5\ 5 #_ #3 #4S> # # #\ #-*-14- !4S>1 2---\---r0rc eZdZUdZejZeed< ede de de e e ffdZ edede de e e ffdZede de de e e ffd Zedede de e e ffd Zd S) JsonStoragez Internal subclass of BaseStorage for the Redis JSON data type. Implements json-specific logic for validation and read/write operations (both sync and async) in Redis. r#r@rAr1cX||d|dS)aSynchronously set a JSON obj in Redis for the given key. Args: client (SyncRedisClient): The Redis client instance. key (str): The key under which to store the JSON obj. obj (Dict[str, Any]): The JSON obj to store in Redis. $N)rsetrEs r.rFzJsonStorage._sets*  #sC(((((r0cKt|ttfr+||d|dS||d|d{VdS)a.Asynchronously set a JSON obj in Redis for the given key. Args: client (AsyncClientOrPipeline): The async Redis client or pipeline instance. key (str): The key under which to store the JSON obj. obj (Dict[str, Any]): The JSON obj to store in Redis. rN)r7rQrRrrrEs r.rHzJsonStorage._asets~ f}.BC D D 3 KKMM  c3 , , , , ,++--##Cc22 2 2 2 2 2 2 2 2 2r0r*cP||S)a,Synchronously retrieve a JSON obj from Redis for the given key. Args: client (SyncRedisClient): The Redis client instance. key (str): The key for which to retrieve the JSON obj. Returns: Dict[str, Any]: The retrieved JSON obj from Redis. )rr{rKs r.rLzJsonStorage._gets {{}}  %%%r0cKt|ttfr'||S||d{VS)aIAsynchronously retrieve a JSON object from Redis for the given key. Args: client (AsyncRedisClient): The async Redis client or pipeline instance. key (str): The key for which to retrieve the JSON object. Returns: Dict[str, Any]: The retrieved JSON object from Redis. N)r7rQrRrr{rKs r.rNzJsonStorage._agetsj f}.BC D D 0;;==$$S)) )**3//////// /r0N)r}rrrrJSONr#rrrrgrrrFrrHrLrNr>r0r.rrs. nD)$$$&)*))4S>)))\) 36 3S 3tCQTH~ 3 3 3\ 3 &* & &c3h & & &\ & 06 0S 0T#s(^ 0 0 0\ 0 0 0r0r)6collections.abcrtypingrrrrrr r r r r pydanticrrredisr redis_versionredis.asyncio.clientrrQredis.asyncio.clusterrrR startswith%redis.commands.search.indexDefinitionr&redis.commands.search.index_definitionrredisvl.exceptionsrredisvl.redis.utilsrredisvl.schemarredisvl.schema.validationr redisvl.typesrrrrrrredisvl.utils.logrredisvl.utils.utilsr r}loggerr"rrr>r0r.rs&&&&&&                        0///////......;:::::IIIIII=C   444444------&&&&&&555555)(((((++++++ H  f&f&f&f&f&)f&f&f&RA-A-A-A-A-+A-A-A-H?0?0?0?0?0+?0?0?0?0?0r0