14164251364
83%

Ran 31 Mar 2025 05:37AM UTC

Jobs 1

Files 57

Run time 1min

Badge

Embed ▾

Committed 28 Mar 2025 10:22AM UTC coverage: 85.526% (+0.06%) from 85.467%

Build # 14164251364

Build Type

push

github

Committed by

web-flow

Commit Message

[FIXED] Duplicate Raft nodes during restart (#6732)

Antithesis triggered duplicate Raft nodes with the following scenario:
- follower needs to catchup based on snapshot, but is leaderless, so
results in: `catchup aborted, no leader`
  - call to `mset.resetClusteredState(..)` attempts to restart Raft node
  - `mset.stop(..)` call resets `sa.Group.node = nil`
  - goes into `js.createRaftGroup(..)` and waits for Raft node to stop
- in the meantime this folllower gets a snapshot from the meta layer,
which updates/checks the stream assignment
  - update notices `sa.Group.node == nil`
- does another call to `js.createRaftGroup(..)`, and also waits for Raft
node to stop
- now Raft node stops and both `js.createRaftGroup(..)` calls duplicate
the Raft node
- health check now notices `Detected stream cluster node skew`, deletes
node and resets

This fix ensures we can't create duplicate Raft nodes for the same
group.

It's unlikely to hit this normally due to requiring exact timing and
leaderless condition on the stream while having a leader for the meta
layer. But was easily reproducible in a test by just calling
`js.createRaftGroup` in parallel.

Signed-off-by: Maurice van Veen <github@mauricevanveen.com>

Run Details

69377 of 81118 relevant lines covered (85.53%)

502761.62 hits per line

Uncovered Existing Lines

Lines	Coverage	∆	File
1	95.0	-0.02%	src/github.com/nats-io/nats-server/server/client.go
2	72.46	0.0%	src/github.com/nats-io/nats-server/server/avl/seqset.go
2	81.28	-0.04%	src/github.com/nats-io/nats-server/server/opts.go
3	90.46	-0.13%	src/github.com/nats-io/nats-server/server/leafnode.go
4	82.63	-0.05%	src/github.com/nats-io/nats-server/server/filestore.go
4	52.61	-0.32%	src/github.com/nats-io/nats-server/server/jetstream_errors_generated.go
4	84.67	-0.04%	src/github.com/nats-io/nats-server/server/stream.go
5	86.48	-0.06%	src/github.com/nats-io/nats-server/server/accounts.go
5	85.94	-0.05%	src/github.com/nats-io/nats-server/server/events.go
7	87.69	0.0%	src/github.com/nats-io/nats-server/server/consumer.go
24	84.65	-0.2%	src/github.com/nats-io/nats-server/server/raft.go
326	82.44	0.6%	src/github.com/nats-io/nats-server/server/jetstream_cluster.go

Jobs

ID	Job ID	Ran	Files	Coverage
1	14164251364.1	31 Mar 2025 05:37AM UTC	57	85.53	GitHub Action Run

nats-io / nats-server / 14164251364
83%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Uncovered Existing Lines

Jobs

Source Files on build 14164251364

nats-io / nats-server / 14164251364 83%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Uncovered Existing Lines

Jobs

Source Files on build 14164251364

nats-io / nats-server / 14164251364
83%

README BADGES
x