kubernetes-sigs / inference-perf / 36063371253
81%

DEFAULT BRANCH: main
Ran
Jobs 1
Files 125
Run time 1min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 80.798%. Remained the same
36063371253

push

github

web-flow 
test: assert reported load metrics against known inputs (#733)

Part of #726. The reported `schedule_delay` and `achieved_rate` are now
asserted to equal the values their inputs imply, instead of only being
checked for shape.

Row in #606: Unit / "Metric accuracy gate, load side (#726)"

## Why this is needed

`summarize_requests` computes `schedule_delay` (`start_time -
scheduled_time`) and `achieved_rate` (`count / send_duration`) at
`inference_perf/reportgen/base.py:720-742`. The only test that reaches
them today is `tests/required/reportgen/test_lifecycle_report_shape.py`,
which builds a fixture with known `scheduled_time` and `start_time`
values and then asserts only `_assert_summary(load["schedule_delay"])`
and `load["achieved_rate"] > 0`. Both numbers are exactly computable
from that fixture and neither is checked, so any arithmetic error in
either survives the suite. `schedule_delay` is the client-side queueing
signal, which is the reading that says whether a reported latency can be
trusted at all, and #564 is what a silently wrong latency-family metric
costs.

## What the test does

- Adds `tests/required/reportgen/test_load_summary_accuracy.py`: 10
tests over `summarize_requests`, which is a pure function, so this is
Unit.
- **Oracle:** the `scheduled_time` and `start_time` values written into
each test. Every expected number is a literal or a one-step arithmetic
expression over those times. Nothing recomputes an expectation by
calling the code under test, so a change to `summarize_requests` cannot
drag the oracle along with it. **Faked:** only the lifecycle metric
objects, built by reusing `_mock_metric` from
`test_lifecycle_report_shape.py` so both modules describe a request the
same way.
- `schedule_delay` cases: all-zero delay, a uniform 0.25s delay, five
mixed delays supplied out of order with mean/min/max/median/p90/p99 all
checked against hand-worked linear interpolation, negative delays from
sends that beat their schedule (the shape ... (continued)

10574 of 13087 relevant lines covered (80.8%)

0.81 hits per line

Jobs
ID Job ID Ran Files Coverage
1 36063371253.1 125
80.8
GitHub Action Run
Source Files on build 36063371253