• Home
  • Features
  • Pricing
  • Docs
  • Announcements
  • Sign In

kubernetes-sigs / inference-perf / 42297162286
87%

Build:
DEFAULT BRANCH: main
Ran 06 Nov 2026 09:40PM UTC
Jobs 1
Files 139
Run time 1min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

06 Nov 2026 09:29PM UTC coverage: 85.918% (+0.02%) from 85.895%
42297162286

push

github

web-flow
Wait for model server readiness before the run starts (#1021)

Part of #979

**Old behavior (main):** there is no readiness wait at all. `main.py`
constructs the model server client and immediately auto-detects the
model, so a server that is still booting fails auto-detection or the
first request instead of being waited for. There is no readiness route,
timeout or CLI flag.

**New behavior:** the wait is always on and lives on the client, not in
configuration. `readiness_timeout_sec` (600 s) sits next to
`readiness_route` as a class attribute on `openAIModelServerClient`, and
a profile whose server boots slower overrides it. Client construction
always waits before model auto-detection.

- vLLM: `readiness_route = "/health"`.
- SGLang and TGI: no dedicated route, so `wait_until_ready()` probes
`GET /v1/models` when `readiness_route` is None.
- A 404 means the route is not served here (llm-d and Envoy answer that
for `/health`); the run logs it and carries on. Connection refused, 503
and every other status keep polling until the budget expires.

How long a server takes to boot never changes a result, so this adds no
config field, constructor argument or CLI flag.

Changes:

- `openAIModelServerClient`: add `readiness_timeout_sec = 600.0`, call
`wait_until_ready()` unconditionally in `__init__`, and let
`wait_until_ready()` probe `/v1/models` when the profile names no route
and return (logging) on a 404.
- The vLLM, SGLang and TGI clients keep their existing constructors;
only vLLM names a readiness route.
- `tests/required/client/modelserver/test_readiness.py` covers the new
behavior: pinned model names still wait, the no-route fallback probes
`/v1/models`, and 404 ends the wait.
- The e2e readiness test and the unit/integration tests that build a
client against a mock, a dead endpoint or an in-process fake server stub
`wait_until_ready`, because it is not the behavior they pin.
- The e2e metric-name checks (`test_vllm_cpu_metric_names` and
`test_sglang... (continued)

32 of 32 new or added lines in 2 files covered. (100.0%)

1 existing line in 1 file now uncovered.

13386 of 15580 relevant lines covered (85.92%)

0.86 hits per line

Coverage Regressions

Lines Coverage ∆ File
1
79.17
-1.04% inference_perf/datagen/base.py
Jobs
ID Job ID Ran Files Coverage
1 42297162286.1 06 Nov 2026 09:40PM UTC 139
85.92
GitHub Action Run
Source Files on build 42297162286
  • Tree
  • List 139
  • Changed 3
  • Source Changed 2
  • Coverage Changed 3
Coverage ∆ File Lines Relevant Covered Missed Hits/Line
  • Back to Repo
  • Github Actions Build #42297162286
  • 7373cc26 on github
  • Prev Build on main (#42133375366)
  • Next Build on main (#42306632534)
  • Delete
STATUS · Troubleshooting · Open an Issue · Sales · Support · CAREERS · ENTERPRISE · START FREE TRIAL · SCHEDULE DEMO
ANNOUNCEMENTS · TWITTER · TOS & SLA · Supported CI Services · What's a CI service? · Automated Testing

© 2026 Coveralls, Inc