kubernetes-sigs / inference-perf / 34476005319

79%

push

github

Add config-driven registry for runtime Prometheus metrics (#611)

Rel: #489

**tldr: derive emitted prometheus metrics set from incoming config**

## Summary

Adds an extensible, config-driven registry for the Prometheus metrics
inference-perf exports about its own runtime (as opposed to metrics
scraped from the model server under test):

- `MetricSpec`: a frozen dataclass declaring one exported metric (name,
docs, type, labels, buckets) plus an `enabled` predicate over the run
`Config` and typed lifecycle hooks (`on_run_start`, `on_request`).
`MetricSpec` is generic over the Prometheus metric type, so a spec whose
hook expects a different metric class than it declares fails `mypy
--strict`.
- `build_metrics(config)`: instantiates only the enabled specs into a
fresh `CollectorRegistry` (never the global default), so disabled
metrics are absent from exposition rather than present at zero.
- `MetricsHub`: holds the instantiated metrics for one run and fans
lifecycle events out to the spec hooks. Hook exceptions are logged and
swallowed (once per spec on the per-request path) so a buggy metric can
never fail the run it is observing.
- Metric definitions live under `observability/metrics/sets/`;
`sets/core.py` holds the always-on specs and future config-conditional
sets get sibling modules aggregated in `sets/__init__.py`.
- `PrometheusMetricsServer` now serves a caller-provided registry
instead of the global default.
- `docs/runtime_metrics.md` is generated from the specs by
`scripts/sync_runtime_metrics_doc.py`, with `pdm run
update:runtime-metrics` / `check:runtime-metrics` to keep it in sync.

Initial metrics:

| Metric | Type | Labels |
| --- | --- | --- |
| `inference_perf_run_elapsed_seconds` | Gauge | none |
| `inference_perf_requests_total` | Counter | `stage`, `status` |

`inference_perf_requests_total` is incremented when a request attempt
completes or fails, not when it is sent.

## Deliberately out of scope (follow-up wiring PR)

- Wiring `Me... (continued)

81 of 83 new or added lines in 3 files covered. (97.59%)

9274 of 11814 relevant lines covered (78.5%)

0.79 hits per line