dcdpr / jp / 26873615790

67%

Build Type

push

github

Committed by web-flow

Commit Message

fix(llm, openai): Stream tool-call events incrementally and add keep-alive (#728)

All four providers (Cerebras, llama.cpp, OpenAI, OpenRouter) now wrap
their event streams with `with_tool_call_keepalive`, injecting a
synthetic heartbeat every 5 seconds while a tool call is in flight. This
prevents the idle-timeout logic from treating a silent gap between
argument chunks as a dead connection.

For the OpenAI Responses API, tool-call events are now emitted
incrementally rather than all at once when the item completes.
`tool_call_start` is fired on `OutputItemAdded`, argument deltas flow
through `FunctionCallArgumentsDelta`, and only the final flush is
emitted on `OutputItemDone`. This opens the tool-call window early so
the keep-alive layer can actually fill the gap while the model is
generating arguments.

The non-streaming fallback path in
`synthesize_non_streaming_output_item_events` is updated to mirror this
three-event sequence (`OutputItemAdded` → `FunctionCallArgumentsDelta`
→ `OutputItemDone`) so that `map_event` produces consistent output
regardless of whether the response was streamed.

---------

Signed-off-by: Jean Mertz <git@jeanmertz.com>

Coverage Stats

27 of 27 new or added lines in 1 file covered. (100.0%)

8 existing lines in 2 files now uncovered.

33672 of 50566 relevant lines covered (66.59%)

250.44 hits per line