IBM / unitxt / 13953172985

81%

push

github

Improve LLM as Judge consistency (#1688)

* Improve final answer prompt to avoid contradictions with the assessment

Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com>

* Be consistent in how the criteria is shown to the LLM

Now both the assessment prompt and the final answer prompt show the criteria in the same way

Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com>

* Set default temperature to 0 to improve consistency

Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com>

* Adapt llm as judge catalog to use temperature = 0

Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com>

---------

Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com>

1569 of 1947 branches covered (80.59%)

Branch coverage included in aggregate %.

9816 of 12243 relevant lines covered (80.18%)

0.8 hits per line