bedrock-kv / bedrock
82%
develop: 82%

LAST BUILD BRANCH: bedrock-5yi/election-flapping
DEFAULT BRANCH: develop
Repo Added
Token s5ZhiY9p3BKcBOY6Au8fohhCDmrp89s9W regen
Build 582 Last
Files 216
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

LAST BUILD ON BRANCH bedrock-5yi/election-flapping
branch: SELECT
CHANGE BRANCH
x
Sync Branches

coverage: 81.654% (-0.007%) from 81.661%
88fdb108976fad36cb3543603a24f670a938959a-PR-329

Pull #329

github

jallum 
bedrock-5yi: Size the coordinator election timeout to the raft heartbeat cadence

Coordinator leadership flapped (roughly one election a second) whenever the
BEAM was under load, most visibly while a coordinator was down and each new
leader's recovery stalled.

bedrock_raft's leader skips a follower it heard from within heartbeat_ms, so
a healthy follower receives AppendEntries every 2 * heartbeat_ms (200ms,
measured on the harness), and a new leader's first AppendEntries can take as
long. The election timeout was 250-350ms: 50ms of slack. Under load a raft
round-trip through the coordinator's DETS log took 50-200ms, stretching
heartbeat gaps to 230-300ms. Of 84 candidacies logged during natural storms
on the harness, 64 fired 254-357ms after the follower's last AppendEntries.
Suspending the leader's coordinator for 150ms started an election in 2-3 of
8 trials; with the timeout below, 0 of 8.

A follower now waits (T, 2T] with T = 5 * heartbeat_ms: the window the
leader itself allows before stepping down when it hears from no quorum of
followers (bedrock_raft's quorum check). The two sides now agree on how long
silence means lost leadership, and a follower tolerates about 300ms of
latency beyond the heartbeat cadence. Failover and single-node boot wait
about 150-650ms longer (501-1000ms instead of 251-350ms).
Pull Request #329: bedrock-5yi: stop stale and too-short election timers from deposing healthy coordinator leaders

8 of 9 new or added lines in 1 file covered. (88.89%)

4 existing lines in 1 file now uncovered.

7188 of 8803 relevant lines covered (81.65%)

1140.97 hits per line

Relevant lines Covered
8803 RELEVANT LINES 7188 COVERED LINES
1140.97 HITS PER LINE
Source Files on develop

Recent builds

Builds Branch Commit Type Ran Committer Via Coverage
88fdb108... bedrock-5yi/election-flapping bedrock-5yi: Size the coordinator election timeout to the raft heartbeat cadence Coordinator leadership flapped (roughly one election a second) whenever the BEAM was under load, most visibly while a coordinator was down and each new leader's reco... Pull #329 jallum github
81.65
a527f4e9... bedrock-1cp/object-storage-resolution bedrock-1cp: Resolve the cluster's object storage backend in one place Four sites found the backend for themselves and disagreed. The coordinator and director read the node config's top level (each with its own path fallback order), the cluster s... Pull #328 jallum github
81.88
8d0bb980... bedrock-w10/leader-durable-core-state bedrock-w10: Recover from the durable bootstrap, not the coordinator's copy A coordinator loaded prior_core_state from the bootstrap once, at boot, and afterwards learned only what its own directors notified it. A follower that later won leadersh... Pull #326 jallum github
81.79
840a00f1... bedrock-hve.1/fsck-chunk-structure bedrock-hve.1: Pin bitstring size variables for Elixir 1.20 Elixir 1.20 warns when a variable is used inside size(...) of a bitstring match without the pin operator, and CI compiles with --warnings-as-errors, so the OTP 29 / Elixir 1.20 job faile... Pull #324 jallum github
81.23
153a2b62... develop bedrock-3t9: seed a fresh cluster's sizing from the node config the durability check validates (#323) Tracks bedrock-3t9. ## Why Strict durability mode is supposed to refuse to start a cluster that can't meet the durability profile — three logs... push web-flow github
81.66
5beb5216... bedrock-3t9/durability-params bedrock-3t9: seed a fresh cluster's sizing from the node config the durability check validates The startup durability check validated desired_logs and desired_replication_factor from node config (durability: [...]), but a fresh cluster's paramete... Pull #323 jallum github
81.67
7e252fae... bedrock-80f/peer-cluster-spike bedrock-80f: Prove a real multi-node cluster can be driven from ExUnit Phase 0 of the chaos-testing harness. Bedrock's identity is node-keyed -- coordinator_nodes is a list of node(), capabilities are per-node config, and ClusterSupervisor refuse... Pull #322 jallum github
81.6
3af051c2... develop Bumped dependencies to clear CVEs push jallum github
81.6
35ae854a... develop bedrock-q67.21.14: Rename components to match their clarified purposes (#304) The q67.21.9 to .13 stack changed what several recovery and distributor components do, and their names kept describing the old behaviour. This is the deferred rename pa... push web-flow github
81.6
e9a9f04e... bedrock-22g/spread-materializer-placement bedrock-22g: address cold-audit notes - materializer_bootstrap_phase: say why recovery's hd() stays — it seats one worker, tag 0, on a fresh cluster, and a singleton has nothing to spread itself against. - server: name the placement cost of t... Pull #290 jallum github
81.52
See All Builds (582)

Badge your Repo: bedrock

We detected this repo isn’t badged! Grab the embed code to the right, add it to your repo to show off your code coverage, and when the badge is live hit the refresh button to remove this message.
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

Refresh