IBM / unitxt / 12926035842

81%

push

github

Add deduplicate operator (#1549)

* Add deduplicate operator

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Deduplicate MMLU

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Update Deduplicate example in documentation for clarity

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Deduplicate social iqa

Signed-off-by: elronbandel <elronbandel@gmail.com>

---------

Signed-off-by: elronbandel <elronbandel@gmail.com>

1420 of 1788 branches covered (79.42%)

Branch coverage included in aggregate %.

9048 of 11428 relevant lines covered (79.17%)

0.79 hits per line