bencheeorg / benchee / 1324

95%

push

travis-ci

Relative statistics (#280)

* Add "relative" statistics to statistics calculation

This is done for 2 reasons:
1. support a new feature I really want to have before 1.0/as a
  little migration bait which is supposed to show the absolute
  difference to the reference scenario/fastest scenario. This
  is good to put performance gains into perspective. It's
  great if something is 100 times as fast, but if it only
  gains me 1 ms in a program that takes 100ms then it's probably
  not worth it
2. centralize the calculation of the multiplicator / lesser / more
  thing so that not every formatter has to implement that
  popular calculation by themselves. This also revealed an
  interesting case that I think we're not handling correctly in
  the formatters yet that happens when the average is 0.

It took more code than I thought it'd but looking at it I also
wouldn't know where to cut it.

Input welcome specifically on the struct field names, half-way
unsure there. I didn't document them yet as the other PR isn't
merged yet.

Other features (squashed because repeatedly resolving conflicts
is tiring):
* test running without any jobs does not blow up
* new fast sample for quicker whole stack dev feedback
* incorporate relative_more instead of impromptu calculations
* Take care of incomparable values

* Handle infinity/non present relative comparison correctly

Added new scenarios to the memory measuyrement brekaer test
because now having to with 0 memory consumption is actually
wanted.

* Display absolute difference for measurements

* Adjust tests for Windows + get rid off subtle time scaling bug

We didn't scale our measured native time for determine_n to
nanoseconds until now :scream:
Impact is little, as to the best of my knowledge it only affects
Windows systems and then only significantly with few runs... still
this shouldn't have happend.

* run sample update

(partly to trigger build again so appveyor hop... (continued)

58 of 58 new or added lines in 5 files covered. (100.0%)

613 of 651 relevant lines covered (94.16%)

13732.13 hits per line