tarantool / tarantool / 7500600609

88%

push

github

replication: replication graceful shutdown

This implies finishing replication TX fibers and stopping applier
threads. This is easy to do using existing applier_stop.

We also need to make sure that there is no client fibers are in
replication code after shutdown. Otherwise we may have difficulties
(assertions) while freeing replication resources. This goal have two
sides. First we have to finish client fibers waiting in replication code
and second we should not allow to wait after shutdonw is done.

Here we probably can achieve first side by just stopping appliers. But in
this case client will get error other than FiberIsCancelled which is
nice to have. So approach is to track client fibers in replication code
and cancel them on shutdown. This approach is also aligned with
iproto/relay shutdown.

There is issue with graceful replication shutdown though. A good example
(on which `replication/shutdown_test.lua` is based) is bootstrapping
a replica with wrong auth in replication URI. In this case applier is
sleeping in reconnect delay and bootstrap code waiting for READY state.
Now comes server shutdown.

Applier is stopped during shutdown and we hit assertion [1]. The issue
is we miss bootstrap fiber notification that applier fiber is cancelled.
That's why the change with `fiber_testcancel` in `applier_f`.

We also drop the assertion in the `replica_on_applier_sync` because
applier can switch to OFF state from any previous state if we cancel
applier fiber.

Part of #8423

[1] Issue assertion stack:
```
  #5  0x00007fe877a54d26 in __assert_fail (
    assertion=0x5637b683b07c "fiber() == applier->fiber",
    file=0x5637b683a03e "./src/box/applier.cc", line=2809,
    function=0x5637b683b05f "void applier_pause(applier*)") at assert.c:101
  #6  0x00005637b62f0f20 in applier_pause (applier=0x5637b7a87a60)
    at /home/shiny/dev/tarantool/src/box/applier.cc:2809
  #7  0x00005637b62f104b in applier_on_state_f (trigger=0x7fe877380a60,
    event=0x5637b7a87a60)... (continued)

66586 of 118922 branches covered (0.0%)

98850 of 113727 relevant lines covered (86.92%)

2678939.19 hits per line