nats-io / nats-server / 31460289290
76%

DEFAULT BRANCH: main
Ran
Jobs 1
Files 67
Run time 1min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 81.104% (+3.9%) from 77.211%
31460289290

push

github

web-flow 
(2.15) [ADDED] Meta peer set rescue for disaster recovery (#8408)

Implements ADR: [Reconfiguring the JetStream meta peer set for disaster
recovery](https://github.com/nats-io/nats-architecture-and-design/pull/418)

This PR adds a `$JS.API.META.RESCUE` endpoint taking `{"quorum_needed":
<number>}` to temporarily lower the number of required servers for
quorum. For example: in a 5-node cluster where 3 nodes are dead and
won't come back, the cluster is stuck and will not be able to elect a
meta leader. The meta rescue command can be used to change quorum to 2
(which automatically expires after 5 minutes). The system is then able
to elect a meta leader, allowing the user to peer-remove the dead
servers. Once the system detects that the cluster's natural
`quorum_needed` arrives at or below the configured rescue
`quorum_needed` this "rescue mode" is automatically stopped and the
cluster is operational again as normal.

> [!WARNING]
> This is ONLY to be used for disaster recovery where it's impossible
for nodes to come back. Using this endpoint allows leader election and
commits with fewer acknowledgements than true quorum, which can
split-brain a cluster if misused/abused. The intended usage is to
request JSZ from the various servers first, which as part of this PR
will now report their cluster size, quorum needed, replicas and whether
they are in `rescue` mode even if there is no meta leader (replicas was
previously limited to just the meta leader). The rescue's
`quorum_needed` should then be set equal to the total number of nodes
that are still alive or can still come back. It can be set lower if the
natural quorum would be lower, but this requires you to be more careful.
Blindly sending `quorum_needed: 1` on a system that has 3 alive servers
to re-form the meta group can result in split-brain.

77108 of 95073 relevant lines covered (81.1%)

368127.6 hits per line

Coverage Regressions

Lines Coverage ∆ File
1151
82.94
 -0.03% src/github.com/nats-io/nats-server/server/jetstream_cluster.go
746
74.86
 0.83% src/github.com/nats-io/nats-server/server/jetstream_api.go
634
80.6
 0.54% src/github.com/nats-io/nats-server/server/raft.go
395
58.52
 0.18% src/github.com/nats-io/nats-server/server/jetstream_errors_generated.go
199
85.82
 1.02% src/github.com/nats-io/nats-server/server/events.go
192
87.16
 16.25% src/github.com/nats-io/nats-server/server/monitor.go
140
84.91
 -0.02% src/github.com/nats-io/nats-server/server/jetstream.go
39
87.42
 -0.49% src/github.com/nats-io/nats-server/server/consumer.go
13
64.61
 1.58% src/github.com/nats-io/nats-server/server/opts.go
4
78.75
 0.06% src/github.com/nats-io/nats-server/server/filestore.go
3
90.9
 -0.38% src/github.com/nats-io/nats-server/server/jetstream_batching.go
3
82.91
 0.12% src/github.com/nats-io/nats-server/server/route.go
2
93.71
 -0.08% src/github.com/nats-io/nats-server/server/gateway.go
2
90.21
 3.79% src/github.com/nats-io/nats-server/server/leafnode.go
2
82.47
 10.06% src/github.com/nats-io/nats-server/server/memstore.go
2
88.73
 2.4% src/github.com/nats-io/nats-server/server/server.go
2
86.66
 0.22% src/github.com/nats-io/nats-server/server/stream.go
Jobs
ID Job ID Ran Files Coverage
1 31460289290.1 67
81.1
 GitHub Action Run
Source Files on build 31460289290