prisma-risk / tsoracle / 26411803505

95%

push

github

fix(driver-paxos): make the single-active leadership lease observable and ABA-proof (#416)

The single-active leadership lease (#403) released only on the consuming
stream's Drop, and surfaced a rejected concurrent subscription through a
single tracing::error!. A server torn down without dropping its stream
(leaked WatchGuard, detached/aborted watch task) leaves the lease held
forever: every subsequent leadership_events() fails closed to an empty
stream and the node stays permanently NotServing, with no metric to alert on.

Fail-closed remains the safety contract. Two Servers built from one
Arc<PaxosDriver> share one OmniPaxos handle and therefore one epoch, so the
epoch fence cannot tell them apart and the lease is the only guard against
overlapping (duplicate) timestamps. Auto-superseding a held lease would
reopen that double-issue window, so recovery stays operator-driven (process
restart) — but the sink is now loud rather than silent.

- Emit tsoracle.leadership_stream.rejected.total on the fail-closed path and
  tag the existing tracing::error! with rejected_generation /
  active_generation, so a wedged lease is an alertable signal.
- Generation-number the lease: AtomicBool becomes a unique-id mint
  (next_generation) plus the holder slot (active_generation). StreamLease now
  releases with a conditional compare_exchange(generation -> 0), freeing the
  slot only if it still owns that generation — making release ABA-proof rather
  than an unconditional clobber.

Document the counter in operations.md. The openraft driver mints leadership
streams freely with no lease; adding single-active protection there is the
separate, larger follow-up noted on #403.

66 of 72 new or added lines in 1 file covered. (91.67%)

9706 of 10080 relevant lines covered (96.29%)

509001.45 hits per line