kubernetes-sigs / inference-perf / 34317783776

80%

push

github

feat(loadgen): bounded retry for pre-response transport faults (#780)

Fixes #777

## Problem

A replayed session is a dependency graph, so a single transport error
doesn't cost one
request — it cancels every event downstream of the failed one. Because
the cost depends on
*where* in the graph the fault landed rather than how serious it was, a
request error rate
well under 1% can erase a double-digit percentage of a run's events.
These are connection
faults, not rejected requests: the endpoint stays up and keeps
answering, it just closes
connections that are in flight or idle in the pool.

There is no retry anywhere in the model-server clients today — a
`ServerDisconnectedError`
goes straight to `process_failure` and the session is marked failed.

## What this does

Adds `load.request_retries` (default **0**) and
`load.request_retry_backoff_sec` (default
0.5, exponential with jitter). Default 0 preserves today's behavior
byte-for-byte.

```yaml
load:
  request_retries: 2
  request_retry_backoff_sec: 0.5
```

## Why only pre-first-byte

The retry boundary is deliberately narrow, and the aiohttp hierarchy
splits along exactly
the line we want:

- **Retried:** `aiohttp.ClientConnectionError` —
`ServerDisconnectedError`, `ClientOSError`.
- **Not retried:** `asyncio.TimeoutError`, even though
`ServerTimeoutError` inherits from
`ClientConnectionError`. A timeout may have reached the model, and
`request_timeout`
  applies **per attempt**, so re-sending multiplies the deadline.
- **Not retried:** `ClientPayloadError` — outside the family on purpose.
A body that broke
mid-transfer already produced a TTFT, so retrying it would report the
second attempt's
  latency instead of the original's.

A `response is None` guard backs the predicate up, so nothing raised
after headers arrive
is retried regardless of type.

What the boundary guarantees is specific: it avoids mixing partial
client-side response
measurements across attempts. It is *not* a claim that the server... (continued)

146 of 163 new or added lines in 8 files covered. (89.57%)

9208 of 11748 relevant lines covered (78.38%)

0.78 hits per line