m iQE\dZddlmZddlZddlmZmZddlmZddl m Z ddl m Z ddl mZdd lmZdd lmZdd lmZdd lmZejeZGd dee ZGddeZGddZGddeeZGddeeZGddeeZ dS)z3Interfaces to be implemented by general evaluators.) annotationsN)ABCabstractmethod)Sequence)Enum)Any)warn) AgentAction)BaseLanguageModel)run_in_executor)ChainceZdZdZdZ dZ dZ dZ dZ dZ dZ d Z d Z d Z d Z d Z dZ dZ dZ dZ dZ dZ dZ dZdS) EvaluatorTypezThe types of the evaluators.qacot_qa context_qapairwise_string score_stringlabeled_pairwise_stringlabeled_score_string trajectorycriterialabeled_criteriastring_distance exact_match regex_matchpairwise_string_distanceembedding_distancepairwise_embedding_distance json_validity json_equalityjson_edit_distancejson_schema_validationN)__name__ __module__ __qualname____doc__QACOT_QA CONTEXT_QAPAIRWISE_STRING SCORE_STRINGLABELED_PAIRWISE_STRINGLABELED_SCORE_STRINGAGENT_TRAJECTORYCRITERIALABELED_CRITERIASTRING_DISTANCE EXACT_MATCH REGEX_MATCHPAIRWISE_STRING_DISTANCEEMBEDDING_DISTANCEPAIRWISE_EMBEDDING_DISTANCE JSON_VALIDITY JSON_EQUALITYJSON_EDIT_DISTANCEJSON_SCHEMA_VALIDATIONC:\Users\Dell Inspiron 16\Desktop\tws\AgrotaPowerBi\back-agrota-powerbi\mcp-client-agrota\venv\Lib\site-packages\langchain_classic/evaluation/schema.pyrrs&& B F%JS'O!L7H1@#VH<)7'OPKIKN9=-M"?;#M.#M=-T5IIr=rc:eZdZdZeed dZdS) LLMEvalChainz,A base class for evaluators that use an LLM.llmr kwargsrreturnc dS)z#Create a new evaluator from an LLM.Nr<)clsrArBs r>from_llmzLLMEvalChain.from_llmNr=N)rAr rBrrCr@)r$r%r&r' classmethodrrFr<r=r>r@r@KsB66222^[222r=r@ceZdZdZeddZeddZeddZeddZ dddZ d S)_EvalArgsMixinz(Mixin for checking evaluation arguments.rCboolcdSz2Whether this evaluator requires a reference label.Fr<selfs r>requires_referencez!_EvalArgsMixin.requires_referenceW ur=cdS)0Whether this evaluator requires an input string.Fr<rNs r>requires_inputz_EvalArgsMixin.requires_input\rQr=strc"d|jjdS)z&Warning to show when input is ignored.zIgnoring input in , as it is not expected. __class__r$rNs r>_skip_input_warningz"_EvalArgsMixin._skip_input_warningasVDN$;UUUUr=c"d|jjdS)z*Warning to show when reference is ignored.zIgnoring reference in rWrXrNs r>_skip_reference_warningz&_EvalArgsMixin._skip_reference_warningfs WT^%< V V V r=N reference str | Noneinput_Nonec&|jr ||jjd}t|||jst |jd|jr ||jjd}t|||jst |jddSdSdS)aTCheck if the evaluation arguments are valid. Args: reference: The reference label. input_: The input string. Raises: ValueError: If the evaluator requires an input string but none is provided, or if the evaluator requires a reference label but none is provided. Nz requires an input string.) stacklevelz requires a reference string.)rTrYr$ ValueErrorr rZrPr\)rOr]r_msgs r>_check_evaluation_argsz%_EvalArgsMixin._check_evaluation_argsms   "6>^,HHHCS// !  d&9  )a 8 8 8 8  " "y'8^,KKKCS// !  )@ -! < < < < < < ! r=rCrKrCrU)NN)r]r^r_r^rCr`) r$r%r&r'propertyrPrTrZr\rfr<r=r>rJrJTs22 XXVVVXV   X !%!=======r=rJceZdZdZeddZeddZedddddZdddddZ dddddZ dddddZ dS)StringEvaluatorzString evaluator interface. Grade, tag, or otherwise evaluate predictions relative to their inputs and/or reference labels. rCrUc|jjS)zThe name of the evaluation.rXrNs r>evaluation_namezStringEvaluator.evaluation_names~&&r=rKcdSrMr<rNs r>rPz"StringEvaluator.requires_referencerQr=Nr]input prediction str | Anyr]str | Any | NonerprBrdictc dS)aEvaluate Chain or LLM output, based on optional input and label. Args: prediction: The LLM or chain prediction to evaluate. reference: The reference label to evaluate against. input: The input to consider during evaluation. **kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: The evaluation results containing the score or value. It is recommended that the dictionary contain the following keys: - score: the score of the evaluation, if applicable. - value: the string value of the evaluation, if applicable. - reasoning: the reasoning for the evaluation, if applicable. Nr<rOrqr]rprBs r>_evaluate_stringsz!StringEvaluator._evaluate_stringsrGr=c@Ktd|jf|||d|d{VS)aAsynchronously evaluate Chain or LLM output, based on optional input and label. Args: prediction: The LLM or chain prediction to evaluate. reference: The reference label to evaluate against. input: The input to consider during evaluation. **kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: The evaluation results containing the score or value. It is recommended that the dictionary contain the following keys: - score: the score of the evaluation, if applicable. - value: the string value of the evaluation, if applicable. - reasoning: the reasoning for the evaluation, if applicable. Nrqr]rp)r rwrvs r>_aevaluate_stringsz"StringEvaluator._aevaluate_stringss`.%   " "             r=r^c R||||jd|||d|S)aEvaluate Chain or LLM output, based on optional input and label. Args: prediction: The LLM or chain prediction to evaluate. reference: The reference label to evaluate against. input: The input to consider during evaluation. **kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: The evaluation results containing the score or value. r]r_ryr<)rfrwrvs r>evaluate_stringsz StringEvaluator.evaluate_stringssQ& ##i#FFF%t% !      r=cbK||||jd|||d|d{VS)aAsynchronously evaluate Chain or LLM output, based on optional input and label. Args: prediction: The LLM or chain prediction to evaluate. reference: The reference label to evaluate against. input: The input to consider during evaluation. **kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: The evaluation results containing the score or value. r|ryNr<)rfrzrvs r>aevaluate_stringsz!StringEvaluator.aevaluate_stringsss& ##i#FFF,T, !            r=rhrg) rqrrr]rsrprsrBrrCrt) rqrUr]r^rpr^rBrrCrt) r$r%r&r'rirmrPrrwrzr}rr<r=r>rkrks '''X'X '+"&      ^ 8'+"&       H!%       >!%         r=rkcbeZdZdZedddddZdddddZdddddZdddddZdS)PairwiseStringEvaluatorzDCompare the output of two models (or two outputs of the same model).NrorqrU prediction_br]r^rprBrrCrtc dS)Evaluate the output string pairs. Args: prediction: The output string from the first model. prediction_b: The output string from the second model. reference: The expected output / reference string. input: The input string. **kwargs: Additional keyword arguments, such as callbacks and optional reference strings. Returns: `dict` containing the preference, scores, and/or other information. Nr<rOrqrr]rprBs r>_evaluate_string_pairsz.PairwiseStringEvaluator._evaluate_string_pairs rGr=cBKtd|jf||||d|d{VS)Asynchronously evaluate the output string pairs. Args: prediction: The output string from the first model. prediction_b: The output string from the second model. reference: The expected output / reference string. input: The input string. **kwargs: Additional keyword arguments, such as callbacks and optional reference strings. Returns: `dict` containing the preference, scores, and/or other information. Nrqrr]rp)r rrs r>_aevaluate_string_pairsz/PairwiseStringEvaluator._aevaluate_string_pairs#sc*%   ' "%            r=c T||||jd||||d|S)rr|rr<)rfrrs r>evaluate_string_pairsz-PairwiseStringEvaluator.evaluate_string_pairsBsT* ##i#FFF*t* !%       r=cdK||||jd||||d|d{VS)rr|rNr<)rfrrs r>aevaluate_string_pairsz.PairwiseStringEvaluator.aevaluate_string_pairs`sv* ##i#FFF1T1 !%             r=) rqrUrrUr]r^rpr^rBrrCrt) r$r%r&r'rrrrrr<r=r>rr sNN !%      ^ 6!%       H!%       F!%         r=rcreZdZdZeddZeddddZddddZddddZ ddddZ dS)AgentTrajectoryEvaluatorz,Interface for evaluating agent trajectories.rCrKcdS)rSTr<rNs r>rTz'AgentTrajectoryEvaluator.requires_inputs tr=N)r]rqrUagent_trajectory!Sequence[tuple[AgentAction, str]]rpr]r^rBrrtc dS)Evaluate a trajectory. Args: prediction: The final predicted response. agent_trajectory: The intermediate steps forming the agent trajectory. input: The input to the agent. reference: The reference answer. **kwargs: Additional keyword arguments. Returns: The evaluation result. Nr<rOrqrrpr]rBs r>_evaluate_agent_trajectoryz3AgentTrajectoryEvaluator._evaluate_agent_trajectoryrGr=cBKtd|jf||||d|d{VS)Asynchronously evaluate a trajectory. Args: prediction: The final predicted response. agent_trajectory: The intermediate steps forming the agent trajectory. input: The input to the agent. reference: The reference answer. **kwargs: Additional keyword arguments. Returns: The evaluation result. N)rqrr]rp)r rrs r>_aevaluate_agent_trajectoryz4AgentTrajectoryEvaluator._aevaluate_agent_trajectorysc,%   + "-            r=c T||||jd||||d|S)rr|rqrprr]r<)rfrrs r>evaluate_agent_trajectoryz2AgentTrajectoryEvaluator.evaluate_agent_trajectorysT, ##i#FFF.t. !-       r=cdK||||jd||||d|d{VS)rr|rNr<)rfrrs r>aevaluate_agent_trajectoryz3AgentTrajectoryEvaluator.aevaluate_agent_trajectorysv, ##i#FFF5T5 !-             r=rg) rqrUrrrprUr]r^rBrrCrt) r$r%r&r'rirTrrrrrr<r=r>rrs66 X!%      ^ :!%       L!%       J!%         r=r)!r' __future__rloggingabcrrcollections.abcrenumrtypingrwarningsr langchain_core.agentsr langchain_core.language_modelsr langchain_core.runnables.configr langchain_classic.chains.baser getLoggerr$loggerrUrr@rJrkrrr<r=r>rs99""""""########$$$$$$------<<<<<<;;;;;;//////  8 $ $3J3J3J3J3JC3J3J3Jl2222252221=1=1=1=1=1=1=1=h~ ~ ~ ~ ~ nc~ ~ ~ Bs s s s s ncs s s l| | | | | ~s| | | | | r=