kubernetes-sigs / inference-perf / 34109861386

79%

Build Type

push

github

Committed by web-flow

Commit Message

feat(datagen): add tool_choice_mode to opt out of replay tool-call forcing (#775)

Fixes #772

### Problem

Replay does not send a recorded turn's tool-choice policy — it computes
and injects
one. When the recorded turn made several calls, or named a tool absent
from that
turn's list, the injected value is `"required"`.

vLLM compiles `"required"` into an unbounded array schema (`minItems:
1`, no
`maxItems`). The grammar only accepts once the array is closed and
nothing biases the
model toward the closing bracket, so it can append calls until
`max_tokens`, repeating
the same call verbatim. Under streaming that truncation is reported as
`finish_reason: "tool_calls"` rather than `"length"`, which is why it
went unnoticed.
Same shape upstream in
[vllm#50399](https://github.com/vllm-project/vllm/issues/50399).

### Change

New `tool_choice_mode` on `SessionReplayConfig` (so both
`otel_trace_replay` and
`weka_trace_replay` inherit it):

```yaml
data:
  otel_trace_replay:
    tool_choice_mode: force_recorded   # default — unchanged behavior
    tool_choice_mode: as_recorded      # inject nothing; the model chooses
```

`as_recorded` gates both injection sites — Chat and Anthropic Messages,
which
inherits the field from the Chat event class.

Forcing stays the default deliberately. A replayed session is a graph:
the successor
turn already holds the `role: "tool"` messages answering this turn's
calls, so a model
that replies in prose leaves those recorded results matching nothing.
`as_recorded`
trades that fidelity for not sending `"required"`.

`tool_choice_mode` is orthogonal to `override_tool_call_max_tokens` — it
changes what
the model *generates*, not the budget the request *asks for*.

Coverage Stats

12 of 12 new or added lines in 2 files covered. (100.0%)

1 existing line in 1 file now uncovered.

9090 of 11627 relevant lines covered (78.18%)

0.78 hits per line