prisma-risk / tsoracle / 26348648998

95%

Build Type

push

github

Committed by web-flow

Commit Message

fix(server): retry fence transient errors while racing the leadership stream (#229) (#235)

A fence that exhausted its bounded TransientDriver retry budget published
NotServing and broke out of the retry loop. A node still elected leader at
the same term then received no fresh leadership event (the leadership stream
dedups identical projections), leaving it NotServing indefinitely — a
cluster-wide timestamp-allocation outage until an operator restarts the
process or forces a leadership transition.

Retry TransientDriver indefinitely instead, racing each capped backoff
against the leadership stream so a genuine leadership change — which some
drivers (e.g. the paxos host, whose append failures map to TransientDriver
with no NotLeader escape) report as a transient error rather than NotLeader —
is observed and dispatched rather than spun past. An event seen mid-retry is
stashed and handled on the next watch iteration.

Add a feature-gated tsoracle.leader_transition.fence_transient_retries.total
counter and a rate-limited warning so a sustained stuck fence is observable,
and add regression tests for both the still-leader recovery and the
mid-retry leadership transition.

Coverage Stats

20 of 24 new or added lines in 1 file covered. (83.33%)

4 existing lines in 1 file now uncovered.

10373 of 10939 relevant lines covered (94.83%)

477745.99 hits per line