michelole / trec-pm

23%

master: 23%

LAST BUILD ON BRANCH stats

branch: SELECT

CHANGE BRANCH

x

• No branch selected
• issue-13
• issue-28
• issue-48
• issue-50
• issue-52
• issue-56
• master
• plots
• stats

Build # 64

Build Type

push

travis-ci-com

Committed by michelole

Commit Message

Add support for paired statistical tests

The current [Python code for statistical testing](https://github.com/JULIELab/trec-pm/blob/master/notebooks/sigir2020-notebooks/testing/StatisticalTests.ipynb) uses the `randtest` library for approximate randomization, which assumes groups are independent.

However, our experimental units (topics) are paired (i.e., we run all experiments on all of them) and therefore there is a dependence among runs.

This commit reimplements statistical testing using the popular R `coin` library on top of the existing code for parsing `trec_eval` and `sampleval` files.

Run Details

1729 of 7571 relevant lines covered (22.84%)

0.46 hits per line