IBM / unitxt / 17129746456 / 1

81%

main: 81%

push

github

Add ReflectionToolCallingMetricSyntactic for evaluating tool call predictions referenceless (#1923)

* Add ReflectionToolCallingMetricSyntactic for evaluating tool call predictions

- Implemented ReflectionToolCallingMetricSyntactic to assess tool calls without references using static checks.
- Added corresponding JSON schema for the metric.
- Created example usage in evaluate_tool_calling_with_reflection.py.
- Updated test_metrics.py to include comprehensive tests for the new metric.

* Install llmevalkit before tests

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Update

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Another try

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Another try

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Another try

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Another try

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Another try

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Another try

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Another try

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Another try

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Update

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Fix style

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Fix examples

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Update the message

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Improve docs

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Update

Signed-off-by: elronbandel <elronbandel@gmail.com>

---------

Signed-off-by: elronbandel <elronbandel@gmail.com>
Co-authored-by: Koren Lazar <koren.lazar@ibm.com>
Co-authored-by: elronbandel <elronbandel@gmail.com>

1595 of 1992 branches covered (80.07%)

Branch coverage included in aggregate %.

10840 of 13396 relevant lines covered (80.92%)

0.81 hits per line