dcdpr / jp / 26763245628

67%

push

github

feat(config, llm): Add stream idle timeout to abort silent connections (#708)

Streaming LLM connections can hang indefinitely if a provider goes
silent mid-response without closing the connection. This adds a
configurable idle timeout that triggers a retryable error when no
activity is received within the specified window.

A new `stream_idle_timeout_secs` field is added to `RequestConfig` with
a default of 60 seconds. Setting it to `0` disables the timeout
entirely. The timer resets on every received item, so only genuinely
silent connections are affected.

When the idle deadline fires, `with_idle_timeout` yields a
`StreamErrorKind::Timeout` error and closes the stream, allowing the
existing retry layer to rebuild and reissue the request. This is
particularly useful for models with long generation pauses (e.g.
deep-research models), where the timeout can be raised to avoid spurious
retries:

```
jp config set assistant.request.stream_idle_timeout_secs 120
```

The implementation wraps any `EventStream` in an `async-stream` loop
that races each `next()` poll against a `tokio::time::timeout`.

Signed-off-by: Jean Mertz <git@jeanmertz.com>

41 of 43 new or added lines in 3 files covered. (95.35%)

32977 of 49878 relevant lines covered (66.12%)

236.78 hits per line