kubernetes-sigs / inference-perf / 35916861843

81%

Build Type

push

github

Committed by web-flow

Commit Message

perf(metrics): optimize multiprocess metric collection (#806)

## What does this PR do?

This PR optimizes multiprocess metric collection in
[`inference_perf/metrics/request_collector/multiprocess.py`](https://github.com/kubernetes-sigs/inference-perf/blob/<a class=hub.com/kubernetes-sigs/inference-perf/commit/<a class="double-link" href="https://git"><a class=hub.com/kubernetes-sigs/inference-perf/commit/8794dc9b8e84cf906377484379ba1b8fcfa3e829">8794dc9b8/inference_perf/metrics/request_collector/multiprocess.py):

1. **Bulk Queue Draining
([`_drain_batch`](https://github.com/kubernetes-sigs/inference-perf/blob/8794dc9b8e84cf906377484379ba1b8fcfa3e829/inference_perf/metrics/request_collector/multiprocess.py#L45-L80))**:
Replaces single-item executor thread calls (`run_in_executor(None,
self.queue.get)`) with a batched drain inside the executor thread (an
initial blocking get with 0.5s timeout followed by non-blocking
`get_nowait()` up to 4,096 items per dispatch).
2. **Batch Queue Ingestion**: Extends the metric list directly via
`batch.extend(...)` and `metrics.extend(batch)`, supporting both
singletons and pre-batched metric lists.
3. **Streamlined Circuit Breaker Feeding**: Feeds metrics to circuit
breakers via `feed_breakers(item)` directly without importing internal
dictionaries.
4. **Unit Test Coverage
([`tests/required/metrics/test_multiprocess_collector.py`](https://github.com/kubernetes-sigs/inference-perf/blob/8794dc9b8e84cf906377484379ba1b8fcfa3e829/tests/required/metrics/test_multiprocess_collector.py))**:
Tests context manager lifecycle, mixed batch and singleton queue
draining, bulk drain capacity, and circuit breaker dispatch using
`unittest.mock.patch.dict`.

## Why is this change needed?

Under high concurrency with multiple worker processes, scheduling an
`event_loop.run_in_executor` invocation for every single metric item
creates a severe asyncio event loop bottleneck:
- **Event loop saturation**: `asyncio._run_once` spent substantial time
dispatching thread pool tasks for individual queue reads, delaying HTTP
request processing and client-side scheduling.
- **Queue polling overhead**: High c... (continued)

Coverage Stats

15 of 36 new or added lines in 1 file covered. (41.67%)

4 existing lines in 1 file now uncovered.

10454 of 12970 relevant lines covered (80.6%)

0.81 hits per line