• Home
  • Features
  • Pricing
  • Docs
  • Announcements
  • Sign In

kubernetes-sigs / inference-perf / 42925197733
87%

Build:
DEFAULT BRANCH: main
Ran 10 Nov 2026 11:51PM UTC
Jobs 1
Files 141
Run time 1min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

10 Nov 2026 11:39PM UTC coverage: 86.283% (+0.02%) from 86.263%
42925197733

push

github

web-flow
feat(observability): keep /metrics up after the run ends (#1049)

Fixes #1047

## Old behavior

The `/metrics` endpoint starts before the run begins and is torn down
immediately in a `finally` block when `perfrunner.run()` returns.

In short-lived benchmark runs (such as Kubernetes Jobs or short
evaluation stages), managed Prometheus scrapers (such as Google Managed
Service for Prometheus / GMP) that scrape on a fixed interval (e.g. 15s
or 30s) frequently miss the end of the run: the last scrape can land up
to one scrape interval before the run finishes, so the final counter
values, error totals, and the final stage end timestamp are lost.

## New behavior

A new `observability.metrics.linger_seconds` option (default `30.0`
seconds) keeps the HTTP `/metrics` endpoint serving after the load
finishes:

1. `deadline = run_end + linger_seconds` is calculated immediately after
`perfrunner.run()` completes.
2. Report generation, saving to file storage, and summary table printing
proceed normally while `/metrics` remains up.
3. Because report generation happens during the linger window, the
linger never delays reports or storage writes. Once reports are done,
`stop_runtime_metrics` waits for any remaining time up to the deadline
before shutting down the server.
4. If `linger_seconds: 0`, the endpoint stops as soon as reports are
saved.
5. If the run is interrupted by `SIGINT` or `SIGTERM`, linger is aborted
immediately.
6. Existing scraping behavior remains 100% compatible and unchanged
(default port `9464` and binding to all interfaces `0.0.0.0` are
preserved).

## Changes

* `inference_perf/config/observability/config.py`: add `linger_seconds:
float = Field(default=30.0, ge=0.0, allow_inf_nan=False)` to
`RuntimeMetricsConfig`.
* `inference_perf/observability/metrics/prometheus.py`: add
`linger_until(deadline, interrupted, clock, sleep)` helper with
`LINGER_POLL_SECONDS = 0.1` so `SIGINT` and `SIGTERM` (which set an
interrupt flag without raising in `LoadGe... (continued)

18 of 20 new or added lines in 3 files covered. (90.0%)

1 existing line in 1 file now uncovered.

13719 of 15900 relevant lines covered (86.28%)

0.86 hits per line

Uncovered Changes

Lines Coverage ∆ File
2
19.92
2.35% inference_perf/main.py

Coverage Regressions

Lines Coverage ∆ File
1
19.92
2.35% inference_perf/main.py
Jobs
ID Job ID Ran Files Coverage
1 42925197733.1 10 Nov 2026 11:51PM UTC 141
86.28
GitHub Action Run
Source Files on build 42925197733
  • Tree
  • List 141
  • Changed 4
  • Source Changed 3
  • Coverage Changed 4
Coverage ∆ File Lines Relevant Covered Missed Hits/Line
  • Back to Repo
  • Github Actions Build #42925197733
  • c1821451 on github
  • Prev Build on main (#42917842637)
  • Next Build on main (#42967643708)
  • Delete
STATUS · Troubleshooting · Open an Issue · Sales · Support · CAREERS · ENTERPRISE · START FREE TRIAL · SCHEDULE DEMO
ANNOUNCEMENTS · TWITTER · TOS & SLA · Supported CI Services · What's a CI service? · Automated Testing

© 2026 Coveralls, Inc