a idZddlmZddlZddlZddlZddlZddlmZddl m Z m Z ddl m Z ddlmZmZmZmZmZmZddlmZmZmZmZmZdd lmZdd lmZdd lm Z dd l!m"Z"m#Z#m$Z$m%Z%ej&e'Z(Gd deZ)GddedZ*GddeZ+GddedZ,GddZ-ee+e,e.fZ/GddeZ0ee0e.fZ1Gdde-Z2d4dZ3d Z4d5d#Z5Gd$d%Z6d6d(Z7d7d+Z8d8d-Z9d9d1Z:eee e j%e e j$gee+e,ffee;e j%e;e j$gee+e,fffZr?r model_configrrLr'r(r)r2r27s? HHH@E0E8#H####8!G!!!!2!%J%%%%: 5666N66665=AOAAAA;59M9999659M9999!E    ):H---L_'"""#"r(r2ceZdZUdZded<dS)EvaluationResultszqBatch evaluation results. This makes it easy for your evaluator to return multiple metrics at once. zlist[EvaluationResult]resultsNr!r'r(r)rOrOas* $###!!r(rOc>eZdZdZe d dd Z d dd ZdS) RunEvaluatorzEvaluator interface class.NrunrexampleOptional[Example]evaluator_run_idOptional[uuid.UUID]rC*Union[EvaluationResult, EvaluationResults]cdS)zEvaluate an example.Nr')rKrSrTrVs r) evaluate_runzRunEvaluator.evaluate_runosr(cKtjfd}tjd|d{VS)z#Evaluate an example asynchronously.ctjdi5cdddS#1swxYwYdS)Nr')rhtracing_contextrZ)current_contextrVrTrSrKsr)_run_with_contextz5RunEvaluator.aevaluate_run.._run_with_contexts#66o66 I I((g7GHH I I I I I I I I I I I I I I I I I Is 7;;N)r]get_tracing_contextasyncioget_running_looprun_in_executor)rKrSrTrVr`r_s```` @r) aevaluate_runzRunEvaluator.aevaluate_runxs022 I I I I I I I I I-//??FWXXXXXXXXXr(NNrSrrTrUrVrWrCrX)r"r#r$r%rrZrer'r(r)rRrRlsm$$&*04 ####^#&*04 Y Y Y Y Y Y Yr(rRcJeZdZUdZded< ded< dZded< dZd ed <dS) ComparisonEvaluationResultzFeedback scores for the results of comparative evaluations. These are generated by functions that compare two or more runs, returning a ranking or other feedback. rr3z'dict[Union[uuid.UUID, str], SCORE_TYPE]scoresNr<r=z6Optional[Union[str, dict[Union[uuid.UUID, str], str]]]r6)r"r#r$r%r&r=r6r'r(r)ririsb  HHH@3333459M99996FJGJJJJr(riceZdZdZ d$d%dZ d&d'dZd(dZd)dZed*dZ d+d,dZ d+d-fd Z d$d.d!Z d/d#Z xZS)0DynamicRunEvaluatoraA dynamic evaluator that wraps a function and transforms it into a `RunEvaluator`. This class is designed to be used with the `@run_evaluator` decorator, allowing functions that take a `Run` and an optional `Example` as arguments, and return an `EvaluationResult` or `EvaluationResults`, to be used as instances of `RunEvaluator`. Attributes: func (Callable): The function that is wrapped by this evaluator. NfuncXCallable[[Run, Optional[Example]], Union[_RUNNABLE_OUTPUT, Awaitable[_RUNNABLE_OUTPUT]]]afuncIOptional[Callable[[Run, Optional[Example]], Awaitable[_RUNNABLE_OUTPUT]]]ct|\}|rt|\}d fd }t||ddlm}|2||||_t |d d |_tj |rE|td ||||_t |d d |_dS|tttttgt f|||_t |d d |_dS) zInitialize the `DynamicRunEvaluator` with a given function. Args: func (Callable): A function that takes a `Run` and an optional `Example` as arguments, and returns a dict or `ComparisonEvaluationResult`. inputsr9rCc||S|d|d\}}}|S)NrSrTgetrr_ traced_inputsprepare_inputss r)process_inputsz4DynamicRunEvaluator.__init__..process_inputssK% $2N 5!!6::i#8#8%% !Q=! r(rrNrzr"rlFunc was provided as a coroutine function, but afunc was also provided. If providing both, func should be a regular function to avoid ambiguity.rrr9rCr9)_normalize_evaluator_funcr langsmithrensure_traceablerogetattr_nameinspectiscoroutinefunction TypeErrorr r rr r_RUNNABLE_OUTPUTrmrKrmrorzrrys @r)__init__zDynamicRunEvaluator.__init__s(";4!@!@~  G&?&F&F #UN ! ! ! ! ! ! d D))))))  $55n6DJ! 4IJJDJ  &t , , J 3 %55^6DJ!z3HIIDJJJ#44XsHW$568HHI4PP-5DI!z3HIIDJJJr(FresultUnion[EvaluationResult, dict]r= uuid.UUID allow_no_keyboolrCr2c\ttrjs|_S stddvr |r |jd<t fddDrtdtdid|iS#t $r}td|d}~wwxYw) NziExpected an EvaluationResult object, or dict with a metric 'key' and optional 'score'; got empty result: r3c3 K|]}|vV dSNr').0krs r) z@DynamicRunEvaluator._coerce_evaluation_result..s'JJq1F?JJJJJJr()r4rr6zrExpected an EvaluationResult object, or dict with a metric 'key' and optional 'score' or categorical 'value'; got r=z[Expected an EvaluationResult object, or dict with a metric 'key' and optional 'score'; got r')rEr2r= ValueErrorrallr)rKrr=res ` r)_coerce_evaluation_resultz-DynamicRunEvaluator._coerce_evaluation_results6 f. / / ' 5'4$M   OFLOOF""|" $ u JJJJ,IJJJJJ  XOUXX$QQ &P&PQQ Q   =4:==  sA B B+B&&B+rPUnion[dict, EvaluationResults]rXcd|vr8|}fd|dD|d<tdi|Stt|dS)NrPc>g|]}|S))r=)r)rrrKr=s r) zBDynamicRunEvaluator._coerce_evaluation_results.. s<..q .NNr(T)r=rr')copyrOrr r9)rKrPr=cps` ` r)_coerce_evaluation_resultsz.DynamicRunEvaluator._coerce_evaluation_resultss   B +ByM%**r** *-- w  }4.   r(MUnion[EvaluationResult, EvaluationResults, dict, str, int, bool, float, list]ct|tr|js||_|St|}|||Sr)rEr2r=_format_evaluator_resultr)rKrr=s r)_format_resultz"DynamicRunEvaluator._format_resultsR f. / / ' 5'4$M)&11..v}EEEr(c"t|dSzCheck if the evaluator function is asynchronous. Returns: bool: `True` if the evaluator function is asynchronous, `False` otherwise. rohasattrrJs r)is_asynczDynamicRunEvaluator.is_async$tW%%%r(rSrrTrUrVrWct|ds_tj}|rt d||||S|tj}d|j i}t|ddrt|j |d<| ||||d}|||S) aEvaluate a run using the wrapped function. This method directly invokes the wrapped function with the provided arguments. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: Union[EvaluationResult, EvaluationResults]: The result of the evaluation. rmtCannot call `evaluate_run` on an async run evaluator from within an running event loop. Use `aevaluate_run` instead.Nr> session_id experimentrun_idr5langsmith_extra)rrbget_event_loop is_running RuntimeErrorrun_until_completereuuiduuid4idrrrrmr)rKrSrTrV running_loopr5rs r)rZz DynamicRunEvaluator.evaluate_run-s"tV$$ Y"133L&&(( Y"R $66t7I7I#w7W7WXXX  ##z|| $3SV#< 3 d + + 9%(%8%8H\ "  '7XNN   ""6+;<<rrrr) rsuperrerrrrrrror)rKrSrTrVr5r __class__s r)rez!DynamicRunEvaluator.aevaluate_runSs&tW%% =..sG<<<<<<<< <  ##z|| $3SV#< 3 d + + 9%(%8%8H\ "zz  '7XNN"         ""6+;<<rrJs r)__repr__zDynamicRunEvaluator.__repr__s4tz4444r(r)rmrnrorp)F)rrr=rrrrCr2)rPrr=rrCrX)rrr=rrCrXrCrrfrg)rSrrTrUrVrW)rSrrTrUrCrXrCr)r"r#r$r%rrrrpropertyrrZrerr __classcell__)rs@r)rlrls7, 8J8J8J8J8J|# <    " F F F F&&&X&&*04 $=$=$=$=$=R&*04 =======D6://///"55555555r(rlrmrnc t|S)zmCreate a run evaluator from a function. Decorator that transforms a function into a `RunEvaluator`. )rlrms r) run_evaluatorrs t $ $$r(i'objrc~t|}t|tkr|dtdz dz}|S)Nz...))reprlen_MAXSIZE)rss r) _maxsize_reprrs: S A 1vv n1 n  & Hr(ceZdZdZ dddZedd Z dddZ dddZ dddZ d dZ e d!dZ d"dZ dS)#DynamicComparisonRunEvaluatorz4Compare predictions (as traces) from 2 or more runs.NrmfCallable[[Sequence[Run], Optional[Example]], Union[_COMPARISON_OUTPUT, Awaitable[_COMPARISON_OUTPUT]]]roUOptional[Callable[[Sequence[Run], Optional[Example]], Awaitable[_COMPARISON_OUTPUT]]]ct|\}|rt|\}d fd }t||ddlm}|2||||_t |d d |_tj |rE|td ||||_t |d d |_dS|tttttt gt"f|||_t |d d |_dS) zInitialize the `DynamicRunEvaluator` with a given function. Args: func (Callable): A function that takes a `Run` and an optional `Example` as arguments, and returns an `EvaluationResult` or `EvaluationResults`. rrr9rCc||S|d|d\}}}|S)NrunsrTrtrvs r)rzz>DynamicComparisonRunEvaluator.__init__..process_inputssK% $2N 6""FJJy$9$9%% !Q=! r(rrNr{r"rlr|r})$_normalize_comparison_evaluator_funcrrrrrorrrrrr r rrr r_COMPARISON_OUTPUTrmrs @r)rz&DynamicComparisonRunEvaluator.__init__s("Fd!K!K~  R&J5&Q&Q #UN ! ! ! ! ! ! d D))))))  $55n6DJ! 4IJJDJ  &t , , J 3 %55^6DJ!z3HIIDJJJ#44!#(9:*,  .5  DI!z3HIIDJJJr(rCrc"t|dSrrrJs r)rz&DynamicComparisonRunEvaluator.is_asyncrr(r Sequence[Run]rTrUrict|ds_tj}|rt d||||Stj}| |}| ||||d}| |||S)zCompare runs to score preferences. Args: runs: A list of runs to compare. example: An optional example to be used in the evaluation. rmrrtagsr) rrbrrrr acompare_runsrr _get_tagsrm_format_results)rKrrTrr=rrs r) compare_runsz*DynamicComparisonRunEvaluator.compare_runsstV$$ "133L&&(( "R $66&&tW55 ~~d##  '4dCC   ##FM4@@@r(cKt|ds|||Stj}||}|||||dd{V}||||S)aEvaluate a run asynchronously using the wrapped async function. This method directly invokes the wrapped async function with the provided arguments. Args: runs (Run): The runs to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: ComparisonEvaluationResult: The result of the evaluation. rorrN)rrrrrror)rKrrTr=rrs r)rz+DynamicComparisonRunEvaluator.acompare_runs s tW%% 4$$T733 3 ~~d##zz  '4dCC"         ##FM4@@@r(c.|||S)aMake the evaluator callable, allowing it to be used like a function. This method enables the evaluator instance to be called directly, forwarding the call to `evaluate_run`. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: ComparisonEvaluationResult: The result of the evaluation. )r)rKrrTs r)rz&DynamicComparisonRunEvaluator.__call__'s  w///r(rcd|jdS)rz>>>>r( list[str]cg}|D]g}|dt|jzt|ddr*|dt|jzh|S)zExtract tags from runs.zrun:rNz experiment:)appendrrrr)rrrSs r)rz'DynamicComparisonRunEvaluator._get_tags<su A AC KKSV, - - -sL$// A MC,?,??@@@ r(r-Union[dict, list, ComparisonEvaluationResult]r=rct|tr|js||_|St|tr$dt ||D|j|d}n8t|t rd|vr |j|d<nd|}t| tdid|i|S#t$r}td||d}~wwxYw)Nc$i|] \}}|j|Sr')r)rrSr4s r) zADynamicComparisonRunEvaluator._format_results..Ss MMMZS%365MMMr()rjr3r=r3zXExpected 'dict', 'list' or 'ComparisonEvaluationResult' result object. Received: result=r=zExpected a dictionary with a 'key' and dictionary of scores mappingrun IDs to numeric scores, or ComparisonEvaluationResult object, got r') rErir=listziprr9rr)rKrr=rmsgrs r)rz-DynamicComparisonRunEvaluator._format_resultsGs: f8 9 9 "' 5'4$M  % % "MM3tV;L;LMMMz!.FF  % % "F"" $ u /%+// S// ! -"M> >J>J>J>J>J@&&&X&AEAAAAA@AEAAAAA:AE00000"????\""""""r(rrrCc t|S)z.Create a comaprison evaluator from a function.)rrs r)comparison_evaluatorrls ) . ..r(r tuple[Union[Callable[[Run, Optional[Example]], _RUNNABLE_OUTPUT], Callable[[Run, Optional[Example]], Awaitable[_RUNNABLE_OUTPUT]]], Optional[Callable[..., dict]]]cdtjdjD}djD|r;t fd|Ds4t fd|Ddkrdd}t |t fd |Dr|d d gkrdfStjr=dfd dfd }tdrtdn|j |_ |fSdfd dfd }tdrtdn|j |_ |fS)N)rSrTrroutputsreference_outputs attachmentsc8g|]\}}|j|jk|Sr'kind VAR_KEYWORDrpnameps r)rz-_normalize_evaluator_func..*XXX(%!-@W@W@W@W@Wr(cHg|]\}}|jtjju| Sr'defaultr Parameteremptyrs r)rz-_normalize_evaluator_func..9 E1 9G-3 3 3  3 3 3r(c3(K|] }|vp|vV dSrr'rrargs_with_defaultssupported_argss r)rz,_normalize_evaluator_func..D  GLE^ # Bu0B'B      r(cg|]}|v| Sr'r'rar s r)rz-_normalize_evaluator_func..$DDDq1C(C(C(C(C(Cr(UInvalid evaluator function. Must have at least one argument. Supported arguments are . Please see https://docs.smith.langchain.com/evaluation/how_to_guides/evaluation/evaluate_llm_application#use-custom-evaluatorsc3(K|] }|vp|vV dSrr'r s r)rz,_normalize_evaluator_func..DCH>5,>#>r(rSrTrrUrCtuple[list, dict, dict]c~|||r|jni|jpi|r |jpini|r |jpinid}i}g}i}jD]i\}}||vr`|j|j|jfvr|||n ||||<|dvrt||n||||<j|||fSN)rSrTrrrrr)rSrT rrrr parametersitemsrPOSITIONAL_OR_KEYWORDPOSITIONAL_ONLYrr rSrTarg_mapkwargsargsrx param_nameparamsigs r)_prepare_inputsz2_normalize_evaluator_func.._prepare_inputs"&07?gnnR"{0b@G#O7#6#<"RBI)Q)>Br  " ),)=)=)?)?%J!W,, :!7!1*!KK (;<<<<181DF:. *-???*'**=>>>!(!4&j1 V]22r(rcDK||\}}}|i|d{VSrr'rSrTr!r rwr%rms r)awrapperz+_normalize_evaluator_func..awrappersH%4OC$A$A!vq!T426222222222r(r"c~|||r|jni|jpi|r |jpini|r |jpinid}i}g}i}jD]i\}}||vr`|j|j|jfvr|||n ||||<|dvrt||n||||<j|||fSrrrs r)r%z2_normalize_evaluator_func.._prepare_inputsr&r(c4||\}}}|i|Srr'r(s r)wrapperz*_normalize_evaluator_func..wrappers0$3OC$A$A!vqtT,V,,,r()rSrrTrUrCr)rSrrTrUrCr r signaturerrrrrrrrr" rmall_argsrr)r,r%r r$r s ` @@@@r)r~r~vsN  D ! !CXXcn&:&:&<&<XXXH,,.. o.      PX     o. DDDDHDDD E E J J G1? G G G ooLT`.    Tz  &t , ,X . 3 3 3 3 3 3> 3 3 3 3 3 3 34,,'j)))&   o. . 3 3 3 3 3 3> - - - - - - - 4,,&j)))%   _- -r(tuple[Union[Callable[[Sequence[Run], Optional[Example]], _COMPARISON_OUTPUT], Callable[[Sequence[Run], Optional[Example]], Awaitable[_COMPARISON_OUTPUT]]], Optional[Callable[..., dict]]]cdtjdjD}djD|r;t fd|Ds4t fd|Ddkrdd}t |t fd |Dr|d d gkrdfStjr=dfd dfd }tdrtdn|j |_ |fSdfd dfd }tdrtdn|j |_ |fS)NrrTrrrrc8g|]\}}|j|jk|Sr'rrs r)rz8_normalize_comparison_evaluator_func.. rr(cHg|]\}}|jtjju| Sr'rrs r)rz8_normalize_comparison_evaluator_func.. rr(c3(K|] }|vp|vV dSrr'r s r)rz7_normalize_comparison_evaluator_func..r r(cg|]}|v| Sr'r'rs r)rz8_normalize_comparison_evaluator_func..rr(rrrc3(K|] }|vp|vV dSrr'r s r)rz7_normalize_comparison_evaluator_func.. rr(rrTrrUrCrcl|||r|jnid|D|r |jpinid}i}g}i}jD]i\}}||vr`|j|j|jfvr|||n ||||<|dvrt||n||||<j|||fS)Nc g|] }|jpi Sr'rrrSs r)rzQ_normalize_comparison_evaluator_func.._prepare_inputs..1BBBc 1rBBBr(r3rrT rrrrrrrrrr rrTrr r!rxr"r#r$s r)r%z=_normalize_comparison_evaluator_func.._prepare_inputs*!&07?gnnRBBTBBBBI)Q)>Br  " ),)=)=)?)?%J!W,, :!7!1*!KK (;<<<<181DF:. *-@@@*'**=>>>!(!4&j1 V]22r(rcDK||\}}}|i|d{VSrr'rrTr!r rwr%rms r)r)z6_normalize_comparison_evaluator_func..awrapperHsH%4OD'$B$B!vq!T426222222222r(r"cl|||r|jnid|D|r |jpinid}i}g}i}jD]i\}}||vr`|j|j|jfvr|||n ||||<|dvrt||n||||<j|||fS)Nc g|] }|jpi Sr'r;r<s r)rzQ_normalize_comparison_evaluator_func.._prepare_inputs..^r=r(r3r>r?r@s r)r%z=_normalize_comparison_evaluator_func.._prepare_inputsWrAr(c4||\}}}|i|Srr'rCs r)r,z5_normalize_comparison_evaluator_func..wrapperus2%4OD'$B$B!vqtT,V,,,r()rrrTrUrCr)rrrTrUrCrr-r/s ` @@@@r)rrsSN  D ! !CXXcn&:&:&<&<XXXH,,.. o,      PX     o, DDDDHDDD E E J J G1? G G G ooLT`,   Tz  &t , ,X , 3 3 3 3 3 3< 3 3 3 3 3 3 34,,'j)))&   _, , 3 3 3 3 3 3< - - - - - - -4,,&j)))%   O+ +r(r;Union[EvaluationResults, dict, str, int, bool, float, list]Union[EvaluationResults, dict]ct|tttfrd|i}n|st d|t|t r1t d|Dst d|dd|i}nBt|trd|i}n(t|trnt d||S) Nr4zdExpected a non-empty dict, str, bool, int, float, list, EvaluationResult, or EvaluationResults. Got c3@K|]}t|tVdSr)rEr9)rxs r)rz+_format_evaluator_result..s,771:a&&777777r(z8Expected a list of dicts or EvaluationResults. Received .rPrzZExpected a dict, str, bool, int, float, list, EvaluationResult, or EvaluationResults. Got ) rErrGrFrrrrr9)rs r)rrs&4,-- 6"   D;A D D    FD ! ! 7777777 T6TTT V$ FC  6" FD ! !   /&, / /    Mr(SUMMARY_EVALUATOR_Tc0dtjdjD}djD|r;t fd|Ds?t fd|Ddkr dd}|r |d |dz }t |t fd |Dr|d d gkrSdfd }tdrtdn|j |_ |S)Nrexamplesrrrrcg|]\}}|Sr'r'rs r)rz0_normalize_summary_evaluator..s===(%===r(cHg|]\}}|jtjju| Sr'rrs r)rz0_normalize_summary_evaluator..rr(c3(K|] }|vp|vV dSrr'r s r)rz/_normalize_summary_evaluator..r r(cg|]}|v| Sr'r'rs r)rz0_normalize_summary_evaluator..rr(rrrLz Received arguments c3 K|]}|vV dSrr')rrr s r)rz/_normalize_summary_evaluator..s(??U.(??????r(rrPSequence[schemas.Run]Sequence[schemas.Example]rCrXcr||d|Dd|Dd|Dd}i}g} jD]E\}}||vr<|j|j|jfvr|||:||||<F|i|}t |tr|St|S)Ncg|] }|j Sr')rrrrTs r)rzA_normalize_summary_evaluator..wrapper..sBBBg7>BBBr(c g|] }|jpi Sr'r;r<s r)rzA_normalize_summary_evaluator..wrapper..s>>>#CK-2>>>r(c g|] }|jpi Sr'r;rZs r)rzA_normalize_summary_evaluator..wrapper..s%T%T%Tgo&;%T%T%Tr(rO) rrrrrrrEr2r) rrPrr r!r"r#rrmr$s r)r,z-_normalize_summary_evaluator..wrappers $BBBBB>>>>>%T%T8%T%T%T GFD%(^%9%9%;%; A A! E((z3-& GJ$78888-4Z-@z*T4*6**F&"233  +F33 3r(r")rrVrPrWrCrX) rr.rrrrrrrr")rmr0rr,r r$r s` @@@r)_normalize_summary_evaluatorr]sSN  D ! !C==cn&:&:&<&<===H,,.. 5      PX     5 DDDDHDDD E E J J C1? C C C   6 5(555 5Coo????h??? ? ?&8PDD  4 4 4 4 4 4 4:*1z)B)B XGD* % % %HX r()rmrn)rr)rmrrCr)rmr rCr)rmr rCr1)rrGrCrH)rmr rCrM)>r% __future__rrbrloggingrabcrcollections.abcrr functoolsrtypingrr r r r r pydanticrrrrrtyping_extensionsrrrr]rlangsmith.schemasrrrr getLoggerr"rHrr+r2rOrRr9rrirrlrrrrrr~rrrrMr]r'r(r)rhs#EE"""""" ////////TSSSSSSSSSSSSS''''''''''''BBBBBBBBBBBB  8 $ $'''''y''' 6 6 6 6 6Ye 6 6 6 6'''''y'''T""""" """"YYYYYYYY8)+>>>>>r(