prisma-risk / tsoracle / 26412469245

95%

Build Type

push

github

Committed by web-flow

Commit Message

fix(server): bound graceful shutdown so a hung driver call can't stall exit (#420)

The user-shutdown arm of `serve_inner` dropped the leader-watch task's
cancel sender and then awaited the task unbounded. But the task observes
its cancel signal only at the `select!` boundaries in
`fence::run_leader_watch` (the event wait and the transient-retry
backoff), never inside a fence attempt. A `load_high_water` /
`persist_high_water` that never returns — the `ConsensusDriver` trait
placed no latency bound — therefore parks the task upstream of any
cancel-observing await, so dropping the sender cannot stop it. Process
exit then blocks until the kubelet escalates to SIGKILL on a drain.

Bound the cooperative wait and abort on overrun. A new
`await_watch_within_grace` waits at most `shutdown_grace` for the task to
stop, then aborts it (emitting `tsoracle.shutdown.watch_aborted.total`
and a warning) so exit always makes progress; the aborted task surfaces
as a cancelled `JoinError`, which `join_to_server_result` maps to
`Ok(())` — the stop was requested. The same bound is applied to the
embedder-facing `WatchGuard::shutdown`, which had the identical unbounded
await; dropping the guard is unchanged (still a detached cooperative
cancel). Because the tonic drain has already completed before this arm
runs, the wait is purely additive and near-instant in the normal case;
the grace only elapses on a genuinely wedged driver.

The grace is configurable via `ServerBuilder::shutdown_grace`, defaulting
to 10s — comfortably under a typical Kubernetes
terminationGracePeriodSeconds. Document the matching driver-trait
contract: implementations must be cancel-safe (a dropped in-flight future
must not corrupt durable state; a late-committing persist is safe under
monotonic-advance) and should prefer returning `TransientDriver` over
blocking indefinitely, since the fence retry loop is cancel-observant.

Regression tests drive a stalled persist through both the `serve_*` and
`Watc... (continued)

Coverage Stats

42 of 44 new or added lines in 1 file covered. (95.45%)

1 existing line in 1 file now uncovered.

10052 of 10407 relevant lines covered (96.59%)

544720.96 hits per line