nats-io / nats-streaming-server / 1871

92%

main: 92%

Pull #813

travis-ci

[FIXED] Clustering: possible deadlock on sub/conn close

There was a wrong assumption that calling raft.Apply() without
waiting on the future result would be non blocking. This is not
the case. RAFT internally uses go channels and there may be cases
when a receiver on a channel is busy and so sending to that channel
will block.

This is what was happening. RAFT would invoke the streaming server
code to apply the replication of a connection close. In this process
we try to remove subscriptions of that connection. This requires
the subStore lock. However, another go routine may be holding that
lock (and possibly subscription's lock, queue state, etc..) and
at the same time call raft.Apply() to replicate an aggregate of
subscription's sent/ack operations. In some cases this would actually
block, causing the deadlock.

Reworked handling of replication of sub's sent/ack events so that
this does not happen anymore.

Resolves #812

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>

131 of 131 new or added lines in 1 file covered. (100.0%)

11393 of 12403 relevant lines covered (91.86%)

46896.41 hits per line