kubernetes-sigs / inference-perf / 36779711671

81%

push

github

fix(metrics): report failed Prometheus queries as missing instead of 0.0 (#845)

Fixes #822

## Summary
Failed Prometheus queries no longer masquerade as real zeros in
`summary_prometheus_metrics.json`. Any failed query (HTTP errors such as
401/403 from a bad or expired `bearer_token`, 5xx, connection refused,
non-`success` status, unparseable or malformed values) is now reported
as
missing (`null`), and a run where every query fails logs an error at the
end instead of hiding in per-query log lines.

## Root cause
`execute_query`
(`inference_perf/client/server_metrics/prometheus_client/base.py`)
returned `0.0` on every failure path, and the inner `execute()` in
`get_model_server_metrics` coerced `None` back to `0.0`. That also
skipped
the #815 colon-name fallback, which only runs when a query returns
`None`.
The result was that with a bad token every metric read 0 and the run
still
succeeded.

## Fix
- `execute_query` returns `None` on all failure paths (also catches
  `TypeError` on value parsing, which previously escaped uncaught, and
  treats valueless result entries as missing instead of silent `0.0`).
- Inner `execute()` propagates `Optional[float]`, so the #815
  underscore-fallback now retries failed queries too.
- `Metric.collect` returns `None` when any of its queries has no usable
result; `ModelServerMetrics` fields are `Optional` (defaults stay zeros,
  so empty and mock metadata behavior is unchanged).
- `summarize_prometheus_metrics` renders missing fields as `null`
(via `_total` and `_summary` helpers; `_ratio` returns `null` if either
  count is missing instead of a bogus `0%`).
- Collection logs a warning when some metrics are missing and an error
  (pointing at url, bearer_token and verify_ssl) when all of them are.

## How it was tested
- New tests (13): failure to `None` for 401, connection error, error
status,
unparseable value and malformed shape; fallback on failure; partial fail
  (`None` field plus warning); all fail (`None` fi... (continued)

43 of 44 new or added lines in 6 files covered. (97.73%)

11392 of 14004 relevant lines covered (81.35%)

0.81 hits per line