• Home
  • Features
  • Pricing
  • Docs
  • Announcements
  • Sign In

kubernetes-sigs / inference-perf / 43934581025
87%

Build:
DEFAULT BRANCH: main
Ran 16 Nov 2026 07:12PM UTC
Jobs 1
Files 142
Run time 1min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

16 Nov 2026 06:59PM UTC coverage: 86.541% (+0.04%) from 86.498%
43934581025

push

github

web-flow
feat(client): add server.wait_for for runs against a pool scaled to zero (#1077)

Fixes #1065.

**Old behavior.** Before the first request the client polls the
readiness route (`/health` for vLLM, `/v1/models` for SGLang and TGI)
until it answers 200 or 404, for up to 600 s. A gateway in front of a
pool that has scaled to zero answers 503 until requests arrive, so the
run fails at the deadline without sending one.

**New behavior.** `server.wait_for` chooses what to wait for:

- `ready` (default): unchanged.
- `reachable`: poll the same route until it gives any HTTP response. A
refused connection, a DNS failure or a connect timeout keeps polling
within the same budget. On the open question: a probe that connects and
then gets no response within the 5 s probe timeout counts as reachable,
since that is a gateway holding the request until a backend exists.
- `none`: no wait.

With `reachable` or `none` the client sends nothing else to the server
before the run. `model_name` is required (config validation rejects
either value without it), the tokenizer loads from
`tokenizer.pretrained_model_name_or_path` or else `model_name` with no
`/v1/models` `root` fallback, and the LoRA adapter check is skipped with
a log line.

Changes:
- `config/client/modelserver/config.py`: `WaitFor`, `server.wait_for`
and the `model_name` rule.
- `openai_client.py`: `wait_until_reachable()` next to
`wait_until_ready()`, which is unchanged. The constructor picks the
wait, and makes the model, tokenizer-root and LoRA lookups only with
`ready`. The vLLM, SGLang and TGI clients and the three factory branches
in `main.py` pass `wait_for` through.
- `docs/config.md`, including that the `reachable` probe is itself a
request, so a gateway that scales up on any request starts its cold
start there; `docs/cli_flags.md` regenerated.
- `test_readiness.py`: per mode, any status ends a `reachable` wait,
refused and connect-timeout probes keep polling, a read timeout counts,
the deadline and pe... (continued)

51 of 51 new or added lines in 5 files covered. (100.0%)

13882 of 16041 relevant lines covered (86.54%)

0.87 hits per line

Jobs
ID Job ID Ran Files Coverage
1 43934581025.1 16 Nov 2026 07:12PM UTC 142
86.54
GitHub Action Run
Source Files on build 43934581025
  • Tree
  • List 142
  • Changed 7
  • Source Changed 6
  • Coverage Changed 7
Coverage ∆ File Lines Relevant Covered Missed Hits/Line
  • Back to Repo
  • Github Actions Build #43934581025
  • 4e0d8750 on github
  • Prev Build on main (#43930557778)
  • Next Build on main (#43937092050)
  • Delete
STATUS · Troubleshooting · Open an Issue · Sales · Support · CAREERS · ENTERPRISE · START FREE TRIAL · SCHEDULE DEMO
ANNOUNCEMENTS · TWITTER · TOS & SLA · Supported CI Services · What's a CI service? · Automated Testing

© 2026 Coveralls, Inc