kubernetes-sigs / inference-perf / 38608881240

86%

push

github

fix: stop a run on SIGTERM the way SIGINT does (#895)

Fixes #894

A SIGTERM ended inference-perf without reports and left its workers
sending requests. Kubernetes stops pods with SIGTERM, so a deleted or
evicted benchmark Job lost its results.

The load generator handles SIGTERM with the SIGINT handler: the current
stage tears down, reports are written and workers stop. Workers fork
after that handler is installed, so each one restores SIGTERM's default
action. Without that, `terminate()`, which teardown uses on wedged
workers, would only set a flag in the worker. The interrupt log line and
span message say "signal".

Teardown still waits up to `stage_teardown_grace_seconds` (default 120
s). With Kubernetes' default `terminationGracePeriodSeconds` of 30 s,
the pod is killed before the reports are written unless the grace is
lowered or the pod's period raised.

Testing:
- New tests: SIGINT and SIGTERM both set the interrupt flag, and
`terminate()` stops a worker forked after the load generator. The first
fails on main; the second fails if workers keep the inherited handler.
- SIGTERM to a 120 s stage against a stub server: exit 0, all 12 workers
gone and reports written, the same as SIGINT. With
`stage_teardown_grace_seconds: 5` it exited in 7 s.
- `pytest tests` passes (2308 passed). So do `ruff check`, `ruff format
--check` and `mypy --strict`.

---------

Signed-off-by: Vu Dinh <vudinh@outlook.com>

12 of 14 new or added lines in 1 file covered. (85.71%)

12084 of 14498 relevant lines covered (83.35%)

0.83 hits per line