kubernetes-sigs / inference-perf / 36818321465

81%

push

github

fix(reportgen): flag ambiguous all-zero kv cache readings via validation.json warning (#846)

## Summary
summary_session_lifecycle_metrics.json reported kv_cache_hit_percent:
0.0 while the server was hitting its prefix cache. Root cause is
upstream of inference-perf: llm-d P/D sidecar normalizes a missing
usage.prompt_tokens_details.cached_tokens to an explicit 0 (thanks
Bslabe123 for pinning it), so an all-zero reading is ambiguous. This PR
keeps the reported 0% number as sent and surfaces the ambiguity as a
warning-severity finding in validation.json instead.

## Root cause
Per the thread: vLLM itself omits prompt_tokens_details when
--enable-prompt-tokens-details is off, but the llm-d sidecar writes
cached_tokens: 0 into the decode response when the prefiller reports
nothing. Usage data alone cannot distinguish that from a genuinely cold
cache, so the number must stay; the signal belongs in validation, not in
a mutated metric or a bare log line.

## Fix
- New SessionLifecycleValidator registered in default_validators():
reads summary_session_lifecycle_metrics.json directly and emits a
session.cache warning when kv_cache_hit_percent is 0 with
sessions_with_cache_info > 0. validate_reports already logs findings,
and e2e tiers assert on validation.json.
- The 0.0 numbers, sessions_with_cache_info, token counts, request-level
prompt_tokens.cached, and CLI output are all unchanged.
- Docs updated in docs/metrics.md.

## How it was tested
- New: tests/reportgen/validation/test_session_validation.py (warning on
all-zero with info; clean on nonzero/None/absent; broken file halts with
a single error).
- Existing: tests/required/reportgen + tests/reportgen/validation +
tests/required/utils -> all pass; ruff check/format clean; mypy --strict
reports no errors in the touched files.
- Note:
tests/required/apis/test_user_session.py::TestLocalUserSessionLifecycle::test_loadgen_mp_does_not_leak_session_context_across_stages
fails identically on clean main in this ... (continued)

33 of 33 new or added lines in 2 files covered. (100.0%)

11427 of 14037 relevant lines covered (81.41%)

0.81 hits per line