databendlabs / openraft / 29088517269

push

github

fix: bound state-machine-worker calls against worker death

# Summary

Calls whose responder lives in the state-machine worker -- `get_snapshot`,
`begin_receiving_snapshot`, `external_state_machine_request`, and
`with_state_machine` -- resolved a send or upgrade failure by joining
`RaftCore` unboundedly. But the worker owns that channel's receiving end
and can die on its own -- e.g. a panic in a state-machine method -- while
`RaftCore` keeps running, so the join never completed and the call hung
forever. They now resolve the real stop cause through a bounded wait.

# Details

- Add `RaftInner::get_core_stop_error_bounded()`, mirroring `recv_msg`'s
  dropped-responder handling: wait up to `RECV_CORE_STOP_TIMEOUT` for
  `RaftCore` to actually stop, then fall back to `Fatal::Stopped`.
- `send_sm_command`, `external_state_machine_request`, and
  `with_state_machine` route every send/upgrade failure through it.
  `external_state_machine_request` previously masked the real cause behind
  a bare `Fatal::Stopped`, contradicting its documented outcomes.
- Add `get_snapshot_returns_stopped_when_sm_worker_dies` and
  `with_state_machine_returns_stopped_when_func_panics`, backed by a
  memstore `panic_on_apply` fault that kills the worker while `RaftCore`
  lives; both assert `Fatal::Stopped` within the bounded wait instead of
  hanging.

15 of 17 new or added lines in 4 files covered. (88.24%)

17474 of 20063 relevant lines covered (87.1%)

156900.44 hits per line