ZǻiN,ddlmZddlmZmZmZmZmZddlZddlm Z ddl m Z m Z dZ dZdd gZd gZd Zd Zd ZdZdZdZdZdZd#dZd$dZiidddf d%dZ d& d'dZ d& d(dZd)dZd*dZd+dZ d,dZ! d- d.dZ"d/dZ# d0 d1dZ$ d2 d3d Z% d4 d5d!Z& d4 d6d"Z'y)7) annotations)AnyDictListOptionalTupleN)Query) ClientErrorCypherTypeError __KGBuilder__ __Entity___Bloom_Perspective_ _Bloom_Scene__Bloom_HAS_SCENE_i' a<CALL apoc.meta.data({sample: $SAMPLE}) YIELD label, other, elementType, type, property WHERE NOT type = 'RELATIONSHIP' AND elementType = 'node' AND NOT label IN $EXCLUDED_LABELS WITH label AS nodeLabel, collect({property:property, type:type}) AS properties RETURN {label: nodeLabel, properties: properties} AS outputa?CALL apoc.meta.data({sample: $SAMPLE}) YIELD label, other, elementType, type, property WHERE NOT type = 'RELATIONSHIP' AND elementType = 'relationship' AND NOT label in $EXCLUDED_LABELS WITH label AS relType, collect({property:property, type:type}) AS properties RETURN {type: relType, properties: properties} AS outputaCCALL apoc.meta.data({sample: $SAMPLE}) YIELD label, other, elementType, type, property WHERE type = 'RELATIONSHIP' AND elementType = 'node' UNWIND other AS other_node WITH * WHERE NOT label IN $EXCLUDED_LABELS AND NOT other_node IN $EXCLUDED_LABELS RETURN {start: label, type: property, end: toString(other_node)} AS outputzCALL apoc.schema.nodes() YIELD label, properties, type, size, valuesSelectivity WHERE type = 'RANGE' RETURN *, size * valuesSelectivity as distinctValueszCALL apoc.meta.graph({sample: 1000, maxRels: 100}) YIELD nodes, relationships RETURN nodes, [rel in relationships | {name:apoc.any.property(rel, 'type'), count: apoc.any.property(rel, 'count')}] AS relationshipscF|jddjddS)zClean string values for schema. Cleans the input text by replacing newline and carriage return characters. Args: text (str): The input text to clean. Returns: str: The cleaned text.    )replace)texts S/opt/lhia/marcimex/agent/venv/lib/python3.12/site-packages/neo4j_graphrag/schema.py_clean_string_valuesrJs" <<c " * *4 55ct|tr|i}|jD]e\}}t|trt|}|$|||<*t|tr't |t ksMt|}|[|||<a|||<g|St|tr8t |t kr%|Dcgc]}t|t|c}Sy|Scc}w)aSanitize the input dictionary or list. Sanitizes the input by removing embedding-like values, lists with more than 128 elements, that are mostly irrelevant for generating answers in a LLM context. These properties, if left in results, can occupy significant context space and detract from the LLM's performance by introducing unnecessary noise and cost. Args: d (Any): The input dictionary or list to sanitize. Returns: Any: The sanitized dictionary or list. N) isinstancedictitems_value_sanitizelistlen LIST_LIMIT)dnew_dictkeyvaluesanitized_valueitems rr r Xs!T'') &JC%&"1%"8#/$3HSME4(u: *&5e&(:Person) Args: driver (neo4j.Driver): Neo4j Python driver instance. is_enhanced (bool): Flag indicating whether to format the schema with detailed statistics (True) or in a simpler overview format (False). database (Optional[str]): The name of the database to connect to. Default is 'neo4j'. timeout (Optional[float]): The timeout for transactions in seconds. Useful for terminating long-running queries. By default, there is no timeout set. sanitize (bool): A flag to indicate whether to remove lists with more than 128 elements from results. Useful for removing embedding-like properties from database responses. Default is False. sample (int): Number of nodes to sample for the apoc.meta.data procedure. Setting sample to -1 will remove sampling. Defaults to 1000. Returns: str: the graph schema information in a serialized format. )r6 is_enhancedr.r+r:sample)get_structured_schema format_schema)r6rAr.r+r:rBstructured_schemas r get_schemarFs2L.  *K 88rcpt|ttttgz|d|||Dcgc]}|d }}t|t t |d|||Dcgc]}|d }}t|ttttgz|d|||Dcgc]}|d } } t|d|||} t|t|||} |D cic] } | d| d c} |D cic] } | d| d c} | | | d d } |rt|| ||| | Scc}wcc}wcc}w#t$rg} g} YfwxYwcc} wcc} w) a( Returns the structured schema of the graph. Returns a dict with following format: .. code:: python { 'node_props': { 'Person': [{'property': 'id', 'type': 'INTEGER'}, {'property': 'name', 'type': 'STRING'}] }, 'rel_props': { 'KNOWS': [{'property': 'fromDate', 'type': 'DATE'}] }, 'relationships': [ {'start': 'Person', 'type': 'KNOWS', 'end': 'Person'} ], 'metadata': { 'constraint': [ {'id': 7, 'name': 'person_id', 'type': 'UNIQUENESS', 'entityType': 'NODE', 'labelsOrTypes': ['Person'], 'properties': ['id'], 'ownedIndex': 'person_id', 'propertyType': None}, ], 'index': [ {'label': 'Person', 'properties': ['name'], 'size': 2, 'type': 'RANGE', 'valuesSelectivity': 1.0, 'distinctValues': 2.0}, ] } } Note: The internal structure of the returned dict depends on the apoc.meta.data and apoc.schema.nodes procedures. Warning: Some labels are excluded from the output schema: - The `__Entity__` and `__KGBuilder__` node labels which are created by the KG Builder pipeline within this package - Some labels related to Bloom internals. Args: driver (neo4j.Driver): Neo4j Python driver instance. is_enhanced (bool): Flag indicating whether to format the schema with detailed statistics (True) or in a simpler overview format (False). database (Optional[str]): The name of the database to connect to. Default is 'neo4j'. timeout (Optional[float]): The timeout for transactions in seconds. Useful for terminating long-running queries. By default, there is no timeout set. sanitize (bool): A flag to indicate whether to remove lists with more than 128 elements from results. Useful for removing embedding-like properties from database responses. Default is False. sample (int): Number of nodes to sample for the apoc.meta.data procedure. Setting sample to -1 will remove sampling. Defaults to 1000. Returns: dict[str, Any]: the graph schema information in a structured format. )EXCLUDED_LABELSSAMPLE)r6r7r8r.r+r:outputzSHOW CONSTRAINTSr6r7r.r+r:label propertiestype) constraintindex) node_props rel_props relationshipsmetadata)r6rEr.r+r:) r?NODE_PROPERTIES_QUERYrHBASE_ENTITY_LABELBASE_KG_BUILDER_LABELREL_PROPERTIES_QUERY EXCLUDED_RELS REL_QUERY INDEX_QUERYr enhance_schema)r6rAr.r+r:rBr2node_propertiesrel_propertiesrSrOrPr=rEs rrCrCs@#'#2$&;<$=      XO$#&'4G     X N ##2$&;<$=      XM"#$    @OOr'{B|$44O=KLrbj"\"22L&#->  /   ] @   PLs/ D  D D $DD.D3 D+*D+c|ddk(rf|jdrU|jdtdztkDrdt|dddSd |dDcgc] }t|c}zS|dd vrO|jd r|jd rd |d d|d S|jdr d|dddSdS|ddk(r,|jdr |dtkDryd|dd|dSycc}w)a Format a single property based on its type and available metadata. Depending on the property type, this function provides either an example value, a range (for numerical and date types), or a list of available options (for strings). If the property is a list that exceeds a defined size limit, it is omitted. Args: prop (Dict[str, Any]): A dictionary containing details of the property, including type, values, min/max, and other metadata. Returns: Optional[str]: A formatted string representing the property details, or None if the property should be skipped (e.g., large lists). rNSTRINGvaluesdistinct_countz Example: "r"zAvailable options: INTEGERFLOATDATE DATE_TIMELOCAL_DATE_TIMEminmaxzMin: z, Max: LISTmin_sizeNz Min Size: z , Max Size: max_size)getDISTINCT_VALUE_LIMITrr#)propr=s r_format_propertyrtws>  F|xDHHX$6 88$&:Q&> ?BV V 4T(^A5F GHJ J&9=hH2*2.HIK  f  88E?txx4;-wtE{m< <8<8JZXq 12!4 RPR R f xx #tJ'7*'DZ 01d:>N=OP P %IsC:c g}|rd|jD]O\}}|jd|d|D]0}t|}||jd|dd|dd|2Q|S|jD]F\}}dj|Dcgc]}|dd |dc}}|j|d |d H|Scc}w) aM Format a collection of properties for nodes or relationships. If `is_enhanced` is True, properties are formatted with additional metadata, such as example values or min/max statistics. Otherwise, they are presented in a more compact form. Args: property_dict (Dict[str, Any]): A dictionary mapping labels (for nodes or relationships) to lists of property definitions. is_enhanced (bool): Flag indicating whether to format properties with additional details. Returns: List[str]: A list of formatted property descriptions. z- **z**z - `property`: rNr, z: z {})rappendrtjoin) property_dictrAformatted_propsrLpropsrsexample props_strs r_format_propertiesrs O)//1 LE5  " "T%#3 4 *40&#**Z 01T&\N!G9M    *//1 ?LE5 CHI4D$%RV ~6II  " "eWC {"#= >  ? JsC c R|Dcgc]}d|dd|dd|ddc}Scc}w)a8 Format relationships into a structured string representation. Args: rels (List[dict]): A list of dictionaries, each containing `start`, `type`, and `end` to describe a relationship between two entities. Returns: List[str]: A list of formatted relationship strings. z(:startz)-[:rNz]->(:end)r/)relsr=s r_format_relationshipsrs;KO OBbG T"V*U2e9+Q ? OO Os$c t|d|}t|d|}t|d}djddj|ddj|ddj|gS)a Format the structured schema into a human-readable string. Depending on the `is_enhanced` flag, this function either creates a concise listing of node labels and relationship types alongside their properties or generates an enhanced, more verbose representation with additional details like example or available values and min/max statistics. It also includes a formatted list of existing relationships. Args: schema (Dict[str, Any]): The structured schema dictionary, containing properties for nodes and relationships as well as relationship definitions. is_enhanced (bool): Flag indicating whether to format the schema with detailed statistics (True) or in a simpler overview format (False). Returns: str: A formatted string representation of the graph schema, including node properties, relationship properties, and relationship patterns. rQrRrSrzNode properties:zRelationship properties:zThe relationships:)rrr{)schemarAformatted_node_propsformatted_rel_propsformatted_relss rrDrDs|(.f\.BKP,VK-@+N*6/+BCN 99  II* + & II) * IIn %    rc g}g} |su|rs|djddkDr\|djdtkrAt|d|d|d|||dd} | jd | d t | || fS|jd |d |d |s| jd|d || fS| jd|dtd|d|| fS)a Build Cypher clauses for string property statistics. Constructs and returns the parts of a Cypher query (`WITH` and `RETURN` clauses) required to gather statistical information about a string property. Depending on property index metadata and whether the query is exhaustive, this function may retrieve a distinct set of values directly from an index or a truncated list of distinct values from the actual nodes or relationships. Args: prop_name (str): The name of the string property. driver (neo4j.Driver): Neo4j Python driver instance. label_or_type (str): The node label or relationship type to query. exhaustive (bool): Whether to perform an exhaustive search or a sampled query approach. prop_index (Optional[List[Any]]): Optional metadata about the property's index. If provided, certain optimizations are applied based on distinct value limits and index availability. database (Optional[str]): The name of the database to connect to. Default is 'neo4j'. timeout (Optional[float]): The timeout for transactions in seconds. Useful for terminating long-running queries. By default, there is no timeout set. sanitize (bool): A flag to indicate whether to remove lists with more than 128 elements from results. Useful for removing embedding-like properties from database responses. Default is False. Returns: Tuple[List[str], List[str]]: A tuple of two lists. The first list contains the `WITH` clauses, and the second list contains the corresponding `RETURN` clauses for the string property. rsizedistinctValuesz&CALL apoc.schema.properties.distinct('z', 'z') YIELD valuerKr'zvalues: z, distinct_count: z'collect(distinct substring(toString(n.`z`), 0, 50)) AS `_values` values: `z _values`[..z], distinct_count: size(`z _values`))rqrrr?rzr") prop_namer6 label_or_type exhaustive prop_indexr.r+r: with_clausesreturn_clausesdistinct_valuess r_build_str_clausesrsOTLN   qM  f % ) qM  . /3G G(!?$ykA       ((:3;O:P Q &  '' 9)E!!* 85   ! !Ii["A B  ''  ! ! {+6J5KL..7[ C   ''rc :d|d|d|d|d }d|d|d}||fS)a] Build Cypher clauses for list property size statistics. Constructs and returns the parts of a Cypher query (`WITH` and `RETURN` clauses) that gather minimum and maximum size information for properties that are lists. These clauses compute the smallest and largest list lengths across the matched entities. Args: prop_name (str): The name of the list property. Returns: Tuple[str, str]: A tuple consisting of a single `WITH` clause (calculating min and max sizes) and a corresponding `RETURN` clause that references these values. z min(size(n.``)) AS `z_size_min`, max(size(n.`z _size_max`z min_size: `z_size_min`, max_size: `r/)r with_clause return_clauses r_build_list_clausesrHsU$ yk)5 k)J @ i[ 7 {*M  %%rc Bg}g}|s3|s1|jd|d|d|jd|d||fS|jd|d|d|jd|d|d |jd |d|d |jd |d |d|d ||fS)aa Build Cypher clauses for numeric and date/datetime property statistics. Constructs and returns the parts of a Cypher query (`WITH` and `RETURN` clauses) needed to gather statistical information about numeric or date/datetime properties. Depending on whether there is an available index or an exhaustive approach is required, this may collect a distinct set of values or compute minimum, maximum, and distinct counts. Args: prop_name (str): The name of the numeric or date/datetime property. exhaustive (bool): Whether to perform an exhaustive search or a sampled query approach. prop_index (Optional[List[Any]]): Optional metadata about the property's index. If provided and the search is not exhaustive, it can be used to optimize the retrieval of distinct values. Returns: Tuple[List[str], List[str]]: A tuple of two lists. The first list contains the `WITH` clauses, and the second list contains the corresponding `RETURN` clauses for the numeric or date/datetime property. zcollect(distinct toString(n.`rrrzmin(n.`z`) AS `z_min`zmax(n.`z_max`zcount(distinct n.`z _distinct`zmin: toString(`z_min`), max: toString(`z_max`), distinct_count: `)rz)rrrrrs r_build_num_date_clausesrds4LN j+I;hyk R   )H=>  '' gi[ {%HIgi[ {%HI  79+Z H  !)-""+-$$-;j:   ''rc |rd|d} nd|d} g} g} i} |s| d|z } |D]}|d}|d}|s/|dd Dcgc]}|d |k(r|d |gk(r |dd k(r|c}nd }|dk(r!t|||||||| \}}| |z } | |z } n\|dvrt|||\}}| |z } | |z } n<|dk(r2t|\}}| j|| j|n|dvrd| j zdz| |<| s| dS| rddj | znd}ddj d| j Dzdz}dj | ||g}|Scc}w) a+ Build a Cypher query for enhanced schema information. Constructs and returns a Cypher query string to gather detailed property statistics for either nodes or relationships. Depending on whether the target entities are below a certain threshold, it may collect exhaustive information or simply sample a few records. This query retrieves data such as minimum and maximum values, distinct value counts, and sample values. Args: driver (neo4j.Driver): Neo4j Python driver instance. structured_schema (Dict[str, Any]): The current schema information including metadata, indexes, and constraints. label_or_type (str): The node label or relationship type to query. properties (List[Dict[str, Any]]): A list of property definitions for the node label or relationship type. exhaustive (bool): Whether to perform an exhaustive search or a sampled query approach. sample_size (int): The number of nodes or relationships to sample when exhaustive is False. Defaults to 5. is_relationship (bool, optional): Indicates if the query is for a relationship type (True) or a node label (False). Defaults to False. database (Optional[str]): The name of the database to connect to. Default is 'neo4j'. timeout (Optional[float]): The timeout for transactions in seconds. Useful for terminating long-running queries. By default, there is no timeout set. sanitize (bool): A flag to indicate whether to remove lists with more than 128 elements from results. Useful for removing embedding-like properties from database responses. Default is False. Returns: str: A Cypher query string that gathers enhanced property metadata. z MATCH ()-[n:`z`]->()z MATCH (n:`z`)z WITH n LIMIT rvrNrTrPrLrMRANGENr`)rr6rrrr.r+r:re)rrrrn)r)BOOLEANPOINTDURATION{ryz RETURN {} AS outputzWITH z, rmzRETURN {rxc32K|]\}}d|d|yw)`rwNr/).0kvs r z-get_enhanced_schema_cypher..s Bdaas#aSMBsz } AS outputr)rrrrzpopr{r)r6rErrMr sample_sizeis_relationshipr.r+r: match_clauserr output_dictrsr prop_typer=r str_w_clauses str_r_clausesnum_date_w_clausesnum_date_r_clauses list_w_clause list_r_clauserr cypher_querys rget_enhanced_schema_cypherrs&Z&}oV< #M?"5 LNK . 66 .B$ L  ,J7@ g;-/|$ 3vJ')     +=#+%%!! , (M= M )L m +N   6M# z6 2  2 . .L 0 0N & +>+S (M=    .  ! !- 0 : : !$~'9'9';!;c!A I].B^ 677=I'JOOL99rK ))Bk.?.?.AB B C   99lKGHL k s"E c V|d}|d}|rtnt} || vry||rdndj|} | syt|||| |tk|||| } |sddgini} t || | ||| d d } | D]!}|d | vs |j | |d #y#t$rYywxYw) a  Enhance the structured schema with detailed statistics for a single node label or relationship type. For the specified node label or relationship type, this function queries the database to gather property statistics such as minimum and maximum values, distinct value counts, and sample values. These statistics are then integrated into the provided structured schema, enriching the schema with more in-depth information about each property. Args: driver (neo4j.Driver): A Neo4j Python driver instance used to run queries against the database. structured_schema (Dict[str, Any]): A dictionary representing the current structured schema, which will be updated with enhanced property statistics. prop_dict (Dict[str, Any]): A dictionary containing the name and count of the node label or relationship type to be enhanced. is_relationship (bool): Indicates whether the properties to be enhanced belong to a relationship (True) or a node (False). database (Optional[str]): The name of the database to connect to. Default is 'neo4j'. timeout (Optional[float]): The timeout for transactions in seconds. Useful for terminating long-running queries. By default, there is no timeout set. sanitize (bool): A flag to indicate whether to remove lists with more than 128 elements from results. Useful for removing embedding-like properties from database responses. Default is False. Returns: None namecountNrRrQ) r6rErrMrrr.r+r:!notifications_disabled_categories UNRECOGNIZED)r6r7r9r.r+r:rrJrv)rYrHrqrEXHAUSTIVE_SEARCH_LIMITr?updater )r6rE prop_dictrr.r+r:rrexcludedr~enhanced_cypherr9 enhanced_inforss renhance_propertiesr sH V D g E /}_H x _k, O S S  E 0+22' O#1>2B C  '!)      =DJ=0 M$z*:;< = s,BB B('B(c t|t|||}|ddD]}t|||d||||ddD]}t|||d|||y) a Enhance the structured schema with detailed property statistics. For each node label and relationship type in the structured schema, this function queries the database to gather additional property statistics such as minimum and maximum values, distinct value counts, and sample values. These statistics are then merged into the provided structured schema dictionary. Args: driver (neo4j.Driver): Neo4j Python driver instance. structured_schema (Dict[str, Any]): The initial structured schema containing node and relationship properties, which will be updated with enhanced statistics. database (Optional[str]): The name of the database to connect to. Default is 'neo4j'. timeout (Optional[float]): The timeout for transactions in seconds. Useful for terminating long-running queries. By default, there is no timeout set. sanitize (bool): A flag to indicate whether to remove lists with more than 128 elements from results. Useful for removing embedding-like properties from database responses. Default is False. Returns: None rKrnodesF)r6rErrr.r+r:rSTN)r?SCHEMA_COUNTS_QUERYr)r6rEr.r+r: schema_countsnoderels rr\r\^s@#! Ma )  /!   Q0  /    r)rstrreturnr)r$rrr)r6 neo4j.Driverr7rr8Dict[str, Any]r9rr. Optional[str]r+Optional[float]r:boolrList[Dict[str, Any]])FNNFi)r6rrArr.rr+rr:rrBintrr)r6rrArr.rr+rr:rrBrrzdict[str, Any])rsrrr)r|rrArr List[str])rrrr)rrrArrr)NNNF)rrr6rrrrrrOptional[List[Any]]r.rr+rr:rrTuple[List[str], List[str]])rrrzTuple[str, str])N)rrrrrrrr)FNNF)r6rrErrrrMrrrrrrrr.rr+rr:rrr)NNF)r6rrErrrrrr.rr+rr:rrNone) r6rrErr.rr+rr:rrr)( __future__rtypingrrrrrneo4jr neo4j.exceptionsr r rWrVrHrYrr#rrrUrXrZr[rrr r?rFrCrtrrrDrrrrrr\r/rrrs#33 9' (/:$%   B?Q 1  6+b %'"#. . . .# .  .  ...f"# .9 .9.9.9 .9  .9  .9 .9f"# L LLL L  L  LL^(V F P P'+"#O(O( O(O( O( $ O(  O(O(O(!O(d&:IM.(.( $.(2E.( .(n!"#s s%ss% s  s  sssss sv##P P%PP P  P  PP Pl## < < %< <  <  <  < r