kubernetes-sigs / inference-perf / 39638590248

85%

push

github

Drop scheduled-ahead requests during stage teardown (#905)

Fixes #902.

When a stage is interrupted, workers can already hold requests whose
scheduled times are still far in the future. Those tasks previously
slept until their scheduled times or until the teardown grace expired,
delaying stage shutdown even though they had not been sent.

This change adds a worker-local draining event and makes the
pre-dispatch schedule wait interruptible. Starting teardown wakes and
retires scheduled-ahead requests immediately, while requests already
sent to the model server still receive the configured grace period. A
deterministic regression test covers one dispatched request plus three
pulled-ahead requests and verifies prompt teardown, skipped-request
accounting, and worker reuse.

Security impact: none.

Validation:

- `.venv/bin/pdm run validate`
- `.venv/bin/pdm run pytest
tests/required/loadgen/test_stage_teardown.py -q` (15 passed)
- `.venv/bin/pdm run check:cov` (81.41%; main baseline 80.90%)
- `.venv/bin/pdm run test` (2314 passed, 10 skipped; local sandbox
prevented socket-binding tests, which passed when rerun with socket
access; two unrelated Rich terminal-history assertions remain
environment-sensitive)

AI-assisted tooling was used for implementation and validation. I
reviewed the diff and test results.

12 of 12 new or added lines in 1 file covered. (100.0%)

12365 of 14783 relevant lines covered (83.64%)

0.84 hits per line