dashbitco / broadway / 0160e33666b8ae8fdfb245bb0ac3b8d5d0a49817-PR-362

93%

main: 93%

Build Type

Pull #362

github

Committed by rslota

Commit Message

Fix graceful shutdown race condition

Currently, when `Topology` is shutting down, it calls `Terminator.trap_exit/1` to allow it to handle `terminate/2` callback and shutdown producers gracefully.
However, `Terminator.trap_exit/1` is using `GenServer.cast/2` which makes it fully asynchronous, making it possible for `Topology`'s
terminate callback to return before `Terminator` starts trapping exits, allowing `Terminator` to be shut down without putting producers into "draining" state.

In my system when we're running ~50-100 Broadway instances we're seeing ~50% of them shutting down properly and the rest of them get stuck with producers
going full blast unaware of the shutdown.

This change simply changes `Terminator.trap_exit/1` to use `GenServer.call/2` instead of `GenServer.cast/2` to make it fully synchronous, which fixed the issue.
Since I don't know Broadway internals at all, please let me know if there is a better way to fix this.

Pull Request Pull Request #362: Fix graceful shutdown race condition

Run Details

2 of 2 new or added lines in 1 file covered. (100.0%)

2 existing lines in 2 files now uncovered.

636 of 684 relevant lines covered (92.98%)

144.42 hits per line