dashbitco / broadway / f52f2f793e16e1c976f5e52adc372457d1445b73

93%

Build Type

push

github

Committed by web-flow

Commit Message

Fix graceful shutdown race condition (#362)

Currently, when `Topology` is shutting down, it calls `Terminator.trap_exit/1` to allow it to handle `terminate/2` callback and shutdown producers gracefully.
However, `Terminator.trap_exit/1` is using `GenServer.cast/2` which makes it fully asynchronous, making it possible for `Topology`'s
terminate callback to return before `Terminator` starts trapping exits, allowing `Terminator` to be shut down without putting producers into "draining" state.

In my system when we're running ~50-100 Broadway instances we're seeing ~50% of them shutting down properly and the rest of them get stuck with producers
going full blast unaware of the shutdown.

This change simply changes `Terminator.trap_exit/1` to use `GenServer.call/2` instead of `GenServer.cast/2` to make it fully synchronous, which fixed the issue.

Run Details

2 of 2 new or added lines in 1 file covered. (100.0%)

2 existing lines in 2 files now uncovered.

636 of 684 relevant lines covered (92.98%)

144.51 hits per line