prisma-risk / tsoracle / 26343988074

97%

Build Type

push

github

Committed by web-flow

Commit Message

fix(server): recover from transient consensus errors during the fence (#227)

The failover fence (`run_leader_watch`) propagated every `ConsensusError`
from `load_high_water`/`persist_high_water` via `?`, terminating the
leader-watch task. Because `serve_with_*` stops the server when that task
ends, a single recoverable error during the volatile post-election window
tore the whole tso server down -- and since the node stays raft-leader, no
survivor could serve either. Standalone deployments recover by process
restart, but embedded users (e.g. the openraft-piggyback example) have no
restart, so a momentary quorum blip or a leadership flap permanently
disabled serving. This surfaced as the intermittent "new leader's
serving-state stream closed before reaching Serving" failure in the
piggyback smoke test.

`ConsensusError` already classifies errors as recoverable
(`NotLeader`/`Fenced`/`TransientDriver`) vs permanent (`PermanentDriver`).
Honor that split in the Leader branch instead of failing fast on all:

  * TransientDriver -- retry the fence with bounded exponential backoff;
    the node is still the elected leader and no fresh leadership event is
    coming to re-drive it. An exhausted budget steps down to NotServing and
    awaits the next event rather than tearing the server down.
  * NotLeader / Fenced -- leadership moved under us; step down to NotServing
    and continue the watch loop so the stream delivers the next state.
  * PermanentDriver / allocator-invariant -- still fatal: propagate so
    into_router poisons serving state and stops serving.

Serving stays NotServing until a fence attempt fully succeeds, so the
"never publish Serving at a stale epoch" invariant the poisoning path
protects holds on every path.

Adds a FaultyDriver test fake plus deterministic regression tests for the
transient-retry and NotLeader-step-down paths, a guard that permanent
errors stay fatal, and rewrites the failpoint test that previously pinned
the fail-stop ... (continued)

Coverage Stats

97 of 101 new or added lines in 2 files covered. (96.04%)

7 existing lines in 2 files now uncovered.

10208 of 10776 relevant lines covered (94.73%)

1458764.13 hits per line