bedrock-kv / bedrock / 88fdb108976fad36cb3543603a24f670a938959a-PR-329

82%

develop: 82%

Build Type

Pull #329

github

Committed by jallum

Commit Message

bedrock-5yi: Size the coordinator election timeout to the raft heartbeat cadence

Coordinator leadership flapped (roughly one election a second) whenever the
BEAM was under load, most visibly while a coordinator was down and each new
leader's recovery stalled.

bedrock_raft's leader skips a follower it heard from within heartbeat_ms, so
a healthy follower receives AppendEntries every 2 * heartbeat_ms (200ms,
measured on the harness), and a new leader's first AppendEntries can take as
long. The election timeout was 250-350ms: 50ms of slack. Under load a raft
round-trip through the coordinator's DETS log took 50-200ms, stretching
heartbeat gaps to 230-300ms. Of 84 candidacies logged during natural storms
on the harness, 64 fired 254-357ms after the follower's last AppendEntries.
Suspending the leader's coordinator for 150ms started an election in 2-3 of
8 trials; with the timeout below, 0 of 8.

A follower now waits (T, 2T] with T = 5 * heartbeat_ms: the window the
leader itself allows before stepping down when it hears from no quorum of
followers (bedrock_raft's quorum check). The two sides now agree on how long
silence means lost leadership, and a follower tolerates about 300ms of
latency beyond the heartbeat cadence. Failover and single-node boot wait
about 150-650ms longer (501-1000ms instead of 251-350ms).

Pull Request Pull Request #329: bedrock-5yi: stop stale and too-short election timers from deposing healthy coordinator leaders

Coverage Stats

8 of 9 new or added lines in 1 file covered. (88.89%)

4 existing lines in 1 file now uncovered.

7188 of 8803 relevant lines covered (81.65%)

1140.97 hits per line