kubernetes-sigs / inference-perf
84%

DEFAULT BRANCH: main
Repo Added
Token NLLSCEurPmbo3BH5LnhZ5rwbNBvlYheVq regen
Build 78 Last
Files 136
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

LAST BUILD ON BRANCH main
branch: SELECT
CHANGE BRANCH
x
Sync Branches

coverage: 84.288% (+0.1%) from 84.139%
39822622994

push

github

web-flow 
test: cover the Prometheus query execution and collection flows (#761)

Fixes #660. Part of #792, Test backfill.

#660 asked for three pins on the Prometheus client: golden PromQL from
the query builder, `execute_query` against canned HTTP responses, and
the `collect_metrics_*` flows against canned query results. #382
delivered the goldens, and main has since gained `execute_query` tests
in `test_prometheus_client.py`. Measured on main,
`tests/required/client/metricsclient` leaves `base.py` at 79%. The
missing lines are the constructor guards, `wait()`, both
`collect_metrics_*` bodies and the None-response guard (34, 44, 51-52,
64-72, 87-103, 185-186).

This adds `test_prometheus_collection.py` and takes `base.py` to 100%.
`execute_query` returns None on every failure it handles, so a failed
query reaches the report as missing rather than as a real-looking zero.
Each of those paths is pinned with its error log. The empty result set
is pinned as the one silent None, since the collector retries None with
the underscore-escaped name. Two bad responses still raise out of report
generation: a 200 whose body is not JSON, and a JSON body that is not an
object. They are pinned in their own class as current behaviour, not
endorsed.

The collection window comes from the stage's own timestamps, `end_time +
scrape_interval + PROMETHEUS_SCRAPE_BUFFER_SEC`. A stage running 50.0 to
100.0 at a 30s interval evaluates at 132.0 over an 82.0 window. Wrong
arithmetic there reads another stage's samples with nothing to flag it.
Two consequences are pinned as documented, not endorsed. A stage
starting within one interval plus buffer of its predecessor's end lands
inside the predecessor's queried range. A FAILED stage is queried
exactly like a COMPLETED one.

Changes:
* `test_prometheus_collection.py`: `execute_query` parsing (including
"0" to 0.0), 6-decimal rounding, first series only, the request it
sends, every None path with its log line, the two raising paths and
NaN/+... (continued)

12462 of 14785 relevant lines covered (84.29%)

0.84 hits per line

Relevant lines Covered
14785 RELEVANT LINES 12462 COVERED LINES
0.84 HITS PER LINE
Source Files on main

Recent builds

Builds Branch Commit Type Ran Committer Via Coverage
39822622994 main test: cover the Prometheus query execution and collection flows (#761) Fixes #660. Part of #792, Test backfill. #660 asked for three pins on the Prometheus client: golden PromQL from the query builder, `execute_query` against canned HTTP respons... push web-flow github
84.29
39819703159 main test: worker count matrix and collector equivalence (#739) Part of #632. Before this PR nothing drove `num_workers` = 0 / 1 / N end to end with real worker processes. Nothing compared `LocalRequestMetricCollector` and `MultiprocessRequestMetricC... push web-flow github
84.14
39819962875 main Run the Hub ShareGPT e2e cases one at a time (#916) Related: #913 The sim suite runs `data_sharegpt_default` under three load shapes. Each case streams `ShareGPT_V3_unfiltered_cleaned_split.json` from the Hub. The file is one 670 MB JSON array, ... push web-flow github
84.14
39817891024 main test: in-process unit tests for the load generator execution core (#704) Part of #659. Tracked in #792 under Test backfill. `Worker.loop`, `run_stage` and `run_session_stage` had no in-process tests. They were only reached through a full `mp_run... push web-flow github
84.13
39817639190 main test: bounded termination and outage recovery (#738) Part of #620 Nothing in the suite asserted that a run against a dead endpoint ends. `LoadConfig.request_timeout` defaults to `None`, so the session gets `aiohttp.helpers.sentinel` instead of a... push web-flow github
83.79
39817354238 main test: pin reported latency metrics to injected values (#734) Part of #726. Tracked in #792 under Test backfill. inference-perf reports four latency metrics per request: time to first token (TTFT), inter-token latency (ITL), time per output token... push web-flow github
83.79
39681093176 main Make the runtime metrics endpoint opt-in (#884) Part of #489 #740 served the runtime metrics endpoint by default and set `host: 0.0.0.0` in the root `config.yml` and the Helm chart values for pod-IP scraping. The root `config.yml` is also what t... push web-flow github
83.7
39638850955 main fix(synthetic_agentic): wire theme system prompt, rotate tools, vary filler (#907) #### What type of PR is this? /kind bug #### What this PR does / why we need it Fixes three realism issues in the synthetic agentic workload generator: 1. **Them... push web-flow github
83.65
39638590248 main Drop scheduled-ahead requests during stage teardown (#905) Fixes #902. When a stage is interrupted, workers can already hold requests whose scheduled times are still far in the future. Those tasks previously slept until their scheduled times or ... push web-flow github
83.64
39337572130 main [URGENT] skip known-bad vLLM releases in the e2e tests (#908) vLLM v0.32.0 shipped on Oct 18 without the fix for its CPU slot-mapping fallback (vllm-project/vllm#61131, fixed on main in #61992 after the release branch was cut). Every vllm-openai-... push web-flow github
83.64
See All Builds (78)