tarantool / luajit / 32949892471

88%

tarantool/master: 93%

push

github

perf: add helper for benchmark results comparison

This patch adds the helper, which is similar to the Google-benchmark
compare.py script [1]. Nevertheless, it has the following semantic
changes:

* compare.py compares the absolute time of the benchmark, which may lead
  to confusion when the time is autotuned to be sure that the benchmark
  runs the minimum required time amount. Hence, this script compares the
  items_per_second metric to check the performance difference even for
  the same absolute times of the benchmarks.
* compare.py compares only 2 files, while this script allows comparing
  directories filled with the same benchmarks (checks intersection with
  warning of unmatched benchmarks), which allows to see the full
  statistic of the patch for the suite.
* This helper doesn't allow running the benchmarks to compare. It
  proceeds only with the given results.
* There is no support for U test.
* --alpha flag allows setting the relative difference threshold from
  which results are considered as changed, instead of the p-value for
  U test.
* The geomean of the benchmarks can be hidden by the flag
  --hide_aggregates, -i.
* The output may be filtered to contain only changed benchmarks by the
  option --changes_only, -c.

For more details, see help in the script.

[1]: https://github.com/google/benchmark/blob/267a11154/tools/compare.py

12756 of 15761 branches covered (80.93%)

Branch coverage included in aggregate %.

21923 of 23560 relevant lines covered (93.05%)

3984732.76 hits per line