kubernetes-sigs / inference-perf / 39817354238
84%

DEFAULT BRANCH: main
Ran
Jobs 1
Files 136
Run time 1min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 83.794% (+0.09%) from 83.7%
39817354238

push

github

web-flow 
test: pin reported latency metrics to injected values (#734)

Part of #726. Tracked in #792 under Test backfill.

inference-perf reports four latency metrics per request: time to first
token (TTFT), inter-token latency (ITL), time per output token (TPOT)
and request latency. The only test that checks them against a known
input is `e2e/tests/test_metrics_fidelity.py` (#614). It needs the
`llm-d-inference-sim` binary and is not in the per-PR unit suite.
Everything in `tests/required` either hands the latency math pre-made
timestamps or only checks that a metric is present.

That is how #564 happened. #410 changed how streamed chunks were
tokenized. ITL was under-reported for about two months before anyone
noticed.

This PR adds 7 tests that inject a known latency profile and check the
reported numbers against it. Most of them:

1. Start a fake OpenAI-compatible server (`FakeOpenAIServer` from #694)
that streams five word-chunks over real HTTP: 600 ms to first token,
then gaps of 400, 100, 400, 100 ms.
2. Run a normal request through the real client, streaming parser,
metric collector and report generator. Only the model server and the
tokenizer are fake.
3. Check TTFT, ITL, TPOT and request latency from the generated report
against the injected pauses.

The reference is the fake server's own per-chunk send timestamp. Server
and client share one process and clock, so each reported number is
compared to when the bytes left, not to how long `asyncio.sleep` was
asked for. Tolerances are 50 ms for delivery and 100 ms for scheduling,
documented in the file.

Changes:

* New `tests/required/integration/test_latency_metric_accuracy.py` with
7 tests:
  * the fake server really pauses, so nothing below passes by accident
  * TTFT matches the 600 ms first-token delay
* changing the first-token delay moves reported TTFT by the same amount
  * ITL min, max, mean and median match the 400/100 ms gaps
* TPOT excludes the first-token wait and is below normalized TPOT
(N... (continued)

12389 of 14785 relevant lines covered (83.79%)

0.84 hits per line

Jobs
ID Job ID Ran Files Coverage
1 39817354238.1 136
83.79
GitHub Action Run
Source Files on build 39817354238