• Home
  • Features
  • Pricing
  • Docs
  • Announcements
  • Sign In

kubernetes-sigs / inference-perf / 40619082179
87%

Build:
DEFAULT BRANCH: main
Ran 28 Oct 2026 02:18AM UTC
Jobs 1
Files 137
Run time 1min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

28 Oct 2026 02:07AM UTC coverage: 85.113%. Remained the same
40619082179

push

github

web-flow
test: side-by-side offered-load parity against vllm bench (#736)

Part of #481. Runs inference-perf and `vllm bench serve` against the
same recording server and asserts the two tools offered the same
workload, which nothing in this repo checks today.

Row in #792 under Test backfill, e2e (deferred from #606's "End-to-end /
Peer-benchmark skew (#481)" row, type `sim`).

Rate and concurrency here are computed by `e2e/utils/load_shape.py` from
#735, so the e2e tier has one definition of each. #735 reads the
client's own timestamps. This reads the server's.

## Why this is needed

Two recent skew reports were first read as inference-perf bugs and both
decomposed into workload non-equivalence (the legacy `range_ratio`
footgun, `0.0` meaning `Uniform[0, len]`; length-sampling, max-model-len
and warmup deltas). Nothing could ask whether the two tools were even
asked for the same thing.

The vllm side never merge-gates, by design. A delta between the tools
can come from an inference-perf regression, a peer-tool change, or
workload non-equivalence. A gate cannot tell which. The inference-perf
side does gate. `pdm run test:e2e` collects this directory, so it runs
in the merge-blocking `e2e-tests` job and checks our own mapping. Peer
results are never asserted on.

## What the test does

- **The oracle is an absorber** (`e2e/utils/absorber.py`), an
OpenAI-compatible aiohttp server that records every request (arrival,
reply-finished, body) while streaming at a configured TTFT and
inter-chunk interval. The recorded requests **are** the offered
workload; no stored baseline.
- **Per tool, against `expected.yaml`:** exact request count,
`max_tokens` multiset, `stream` and `ignore_eos` flags, per-request
prompt tokens under the shared tokenizer (with tolerance, plus a
declared outlier share for the vllm sampler), and either delivered rate
within `rate_tolerance(n, arrival)` or delivered concurrency via
`assert_delivered_concurrency` (never above C, time-weighted plate... (continued)

12927 of 15188 relevant lines covered (85.11%)

0.85 hits per line

Jobs
ID Job ID Ran Files Coverage
1 40619082179.1 28 Oct 2026 02:18AM UTC 137
85.11
GitHub Action Run
Source Files on build 40619082179
  • Tree
  • List 137
  • Changed 0
  • Source Changed 0
  • Coverage Changed 0
Coverage ∆ File Lines Relevant Covered Missed Hits/Line
  • Back to Repo
  • Github Actions Build #40619082179
  • e1c8fbe0 on github
  • Prev Build on main (#40542071612)
  • Next Build on main (#40727487205)
  • Delete
STATUS · Troubleshooting · Open an Issue · Sales · Support · CAREERS · ENTERPRISE · START FREE TRIAL · SCHEDULE DEMO
ANNOUNCEMENTS · TWITTER · TOS & SLA · Supported CI Services · What's a CI service? · Automated Testing

© 2026 Coveralls, Inc