dcdpr / jp / 26762686162

67%

Build Type

push

github

Committed by web-flow

Commit Message

fix(llm, cerebras, llamacpp): Handle pre-completion stream errors (#707)

Both the `cerebras` and `llamacpp` providers shared the same bug: the
`take_while(|e| e.is_ok())` guard dropped the first error from the SSE
stream without letting it reach the handler. That meant a dropped or
stalled connection before `[DONE]` was silently swallowed instead of
being surfaced to the retry layer.

At the same time, the benign connection-close that `EventSource` emits
after `[DONE]` was reaching the retry layer as a spurious error,
triggering unnecessary retries on what is actually a clean end of
stream.

The fix introduces a `finished` flag on `StreamState`. Once the `[DONE]`
sentinel is processed and `Event::Finished` is emitted, `finished` is
set to `true`. The `take_while` guard is replaced with a `seen_error`
latch that lets exactly one error through (so the handler can inspect
it), and the `Err` arm of `handle_sse_event_sync` now drops the error
silently when `finished` is `true` and propagates it otherwise.

`EventSource`'s built-in auto-reconnect is also disabled via
`set_retry_policy(Box::new(Never))` on both providers, since reconnect
logic is owned by the outer retry layer and the internal reconnect was
masking real failures.

The stream-assembly logic is extracted into a standalone
`assemble_event_stream` function that accepts a generic `Stream`, making
it independently testable. Two async tests are added to each provider to
pin the contract:

- `surfaces_stream_error_before_completion`: a transport error before
`[DONE]` must reach the caller as a `StreamError`.
- `swallows_stream_error_after_completion`: the connection close after
`[DONE]` must not surface as an error and the stream must end with
`Event::Finished`.

Signed-off-by: Jean Mertz <git@jeanmertz.com>

Coverage Stats

71 of 73 new or added lines in 2 files covered. (97.26%)

2 existing lines in 1 file now uncovered.

32940 of 49839 relevant lines covered (66.09%)

258.08 hits per line