kubernetes-sigs / inference-perf / 36902545652

81%

push

github

fix(client): drop embedding vectors from stored response bodies (#851)

Fixes #848

Every request's `RequestLifecycleMetric` keeps the full response body in
`response_data`. For embeddings, that body is mostly the vectors, which
nothing reads, so memory grows with every request.

This keeps only `model` and `usage` from a successful embeddings
response (`error is None`). A failed request's body is stored whole,
since it carries the server's error message. That includes a 200
recorded as a failure, such as an in-band error payload once #744 lands.
Other API types are unchanged.

**User-facing change:** in the per-request report, the `response` field
for a successful embeddings request now shows only `model` and `usage`.

Testing:
- The embeddings round-trip test checks that the stored body has only
`model` and `usage`.
- New test: an embeddings error response (400) is stored unchanged.
- New test: a 200 that `process_response` rejects is recorded as a
failure and its body is stored unchanged. It fails with the previous
`status == 200` gate (the body comes back as `{}`).
- `pytest tests/required`, `ruff check`, `ruff format --check` and `mypy
--strict ./inference_perf ./tests` pass.

7 of 10 new or added lines in 1 file covered. (70.0%)

11434 of 14047 relevant lines covered (81.4%)

0.81 hits per line