kubernetes-sigs / inference-perf / 39819703159
85%

DEFAULT BRANCH: main
Ran
Jobs 1
Files 136
Run time 1min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 84.139% (+0.006%) from 84.133%
39819703159

push

github

web-flow 
test: worker count matrix and collector equivalence (#739)

Part of #632.

Before this PR nothing drove `num_workers` = 0 / 1 / N end to end with
real worker processes. Nothing compared `LocalRequestMetricCollector`
and `MultiprocessRequestMetricCollector` aggregates for the same load
either. Both are now asserted.

Tracked in #792 under Test backfill, as one Integration row and one e2e
row.

`num_workers` selects between two execution architectures. Every
motivating bug is process machinery: #589 (datagen not picklable), #590
(crash only on the `num_workers = 0` path, fixed in #613), #469 (barrier
hang), #593 (no diagnostics on worker crash). The default is `max(1,
cpu_count())`, so CI exercises whichever cell the runner has. #526 and
#305 change spawn semantics here and need a gate.

Changes:

* `tests/required/integration/test_worker_matrix.py` (per-change). Cells
0, 1 and 4 each run two stages against `FakeOpenAIServer` over loopback.
Only the server is faked. Worker processes, the stage barrier, the
shipped OpenAI client and its SSE parsing are real. Only the client
constructor is swapped.
* The oracle is the fake server's own record of what it served. In every
cell the requests the client accounts for must equal the requests the
server handled. The mp cells must send exactly 8 per stage. The
in-process cell sends 7 or 8, see below.
* Collector equivalence. The same metric objects go through both
collectors, the multiprocess one fed from a real child process. Exact
for counts and token values. Structural for which latency fields are
`None`. Within 1e-9 for latency summaries, since the multiprocess
collector reassembles them in arrival order.
* Start-method agnostic, so it passes before and after #526. It asserts
directly that the worker payload survives a pickle round trip (#589).
Real-process work runs under `asyncio.wait_for`, so a barrier regression
(#469) fails instead of hanging CI. N is pinned at 4.
* `e2e/tests/test_worker_matrix_smoke.py`... (continued)

12440 of 14785 relevant lines covered (84.14%)

0.84 hits per line

Jobs
ID Job ID Ran Files Coverage
1 39819703159.1 136
84.14
GitHub Action Run
Source Files on build 39819703159