IntelPython / dpnp / 38565753201

83%

push

github

Stop timing NumPy in the ASV benchmarks (#3095)

This PR removes the `numpy` executor from the ASV benchmark suites, so
`bench_elementwise.py`, `bench_linalg.py` and `bench_random.py` now time
only dpnp.

The suites track dpnp performance from commit to commit, and nothing
compares the dpnp and NumPy timings with each other. A dpnp change
cannot move a NumPy timing, so those points carry no signal. They did
have two costs:
- They made up half of the timed points in these three suites (470 of
940).
- Machine noise on them was reported as a dpnp regression. A `master`
commit that only bumped pre-commit hooks was flagged solely because of
`MatMul.time_matmul('numpy', 16, 'float64')` at 1.22x. On a recent PR,
two of the three reported regressions were NumPy points, including
`MatMul.time_dot('numpy', 64, 'float32')` at 1.6x, with the same NumPy
and MKL builds on both sides.

The change keeps the recorded results comparable:
- `executor` stays as a single-value axis (`'dpnp'`), so result names
such as `time_dot('dpnp', 64, 'float32')` are unchanged.
- The `setup` and `time_*` methods are not modified, including the
`numpy` branches some `setup` methods no longer use. ASV versions a
benchmark by the source of these methods, so editing them would make
every result incomparable with the recorded baseline. All 29 such
methods are byte-identical to `master`.

Against a baseline recorded before this change, the NumPy points exist
only on the baseline side. `asv compare` treats them as skipped, so
`--only-changed` hides them and they are not counted as regressions.

`benchmarks/README.md` is updated to match and gets a new "NumPy is not
timed" section explaining the above.

1608 of 3000 branches covered (53.6%)

Branch coverage included in aggregate %.

27915 of 34528 relevant lines covered (80.85%)

19608.66 hits per line