kubernetes-sigs / inference-perf / 39285023412

85%

push

github

feat(trace-replay): replay recorded images as images (#900)

Fixes #899

Trace replay didn't send recorded images as images. What happened
depended on where the image was:
- **In a tool result** (e.g. an agent reading a screenshot), the image
was sent as base64 text, which costs roughly one token per byte of the
image file. The prompt grew far beyond what was recorded, and a few
screenshots could exceed the context window.
- **Anywhere else** (an image block in a user message, an `image_url`
part, an OTel `blob` part), the image was dropped, so the prompt was
smaller than recorded.

Recorded images now replay as image parts, in their recorded position,
with the same bytes on every call:

| Recorded API | Where the image is |
|---|---|
| Anthropic Messages | `image` blocks in user messages and inside
`tool_result` |
| Chat Completions | `image_url` parts in any message, including `role:
"tool"` |
| Responses | `input_image` in `message` and `function_call_output`;
computer use's `computer_call` / `computer_call_output`, replayed as
calls to a `computer` function |
| OTel GenAI | `blob` / `uri` parts |

The `anthropic_messages` renderer sends images natively, including
inside `tool_result`. Images are not resized, pruned or moved: the trace
already reflects the client's image policy. Any recorded API can be
replayed with either `api.type`; with `chat`, a tool-result image stays
in the `role: "tool"` message, where the server's chat template decides
whether the model sees it. Text-only traces replay unchanged.

Also in this change:
- **Claude Code raw-body logs.** A streamed Anthropic request whose
recorded response is one assembled JSON `message` is accepted; a
truncated stream or an error body is still skipped. Claude Code's
`OTEL_LOG_RAW_API_BODIES` files record streamed calls this way.
- **Missing-image warning.** A trace that no longer carries an image's
bytes logs one warning with the count. This covers redacted images,
`[image]` / Langfuse placeho... (continued)

253 of 264 new or added lines in 8 files covered. (95.83%)

12360 of 14778 relevant lines covered (83.64%)

0.84 hits per line