databendlabs / openraft / 24497325976

86%

Build Type

push

github

Committed by drmingdrmer

Commit Message

fix: clear replication backoff after success in stream session

Thanks to @AustenSchunk for reporting this bug (#1723) with a
precise root-cause analysis that made the fix straightforward.

Cause:

Backoff state was split across two pieces that were updated in
different scopes and never coordinated:

  1. `ReplicationCore::backoff_rank` — a counter incremented on
     every RPC error and reset to 0 on every successful response.
     Both updates happen inside the stream response-handling loop.

  2. `Arc<Mutex<Option<Backoff>>>` — the shared `Backoff` iterator
     sampled by `StreamState::backoff_if_enabled` before each
     request. This was only created by `enable_backoff()` and
     only cleared by `disable_backoff()`, both called from the
     outer main loop.

In `LogsSince` (pipeline) mode the outer main loop does not
iterate while a stream session is alive — `handle_response_stream`
consumes responses in a `while let` loop while the request-stream
generator concurrently emits requests. `disable_backoff` therefore
never runs between sessions. So after a transient error spike
followed by recovery:

  - `backoff_rank` is reset to 0 on the first successful response.
  - The `Backoff` object remains `Some(...)` indefinitely.
  - Every subsequent request sleeps for ~200ms before being sent.
  - The throttle persists until the session ends for unrelated
    reasons (network error, cancellation, higher vote).

Fix:

Encapsulate both pieces behind a new `BackoffState` type that
maintains the invariant `rank == 0 => inner == None`. `on_success`
now clears both `rank` and the shared iterator in one place, so
reconciliation no longer depends on the outer loop. `StreamState`
receives a read-only `BackoffConsumer` handle that can only sample
the next delay — it cannot enable, disable, or replace the
backoff, so the ownership of these transitions is clear in the
type system.

Changes:
- Add `BackoffState` owning both the rank counter and the shared
  `Backof... (continued)

Coverage Stats

105 of 106 new or added lines in 4 files covered. (99.06%)

7 existing lines in 3 files now uncovered.

16408 of 18966 relevant lines covered (86.51%)

119081.72 hits per line