dashbitco / broadway

93%

main: 93%

LAST BUILD ON BRANCH fix-graceful-shutdown

branch: fix-graceful-shutdown

CHANGE BRANCH

x

Reset

Sync Branches

• fix-graceful-shutdown
• 225-skipped-with-dummy-producer
• 245-adding-context-to-telemtry-events
• FEATURE/add-configurable-ack
• acco/add-reset-opt-to-update_rate_limiting
• adding_supervisor_init_telemetry_event
• al/1.18
• al/minor-doc-improvements
• al/more-types
• al/rename
• al/update-deps-and-ci
• al/update-elixir
• andrea/fix-process-name-callback
• andrea/more-nimble-options
• andrea/nimble
• andrea/rpdoucers
• andrea/speed-up-ci
• byte-size-batching
• cg/improve-prepare-docs
• change-failed-typespec
• code-coverage-on-coveralls
• concurrency_setting_notes
• default-batcher
• demand-dispatcher-shuffle-demands
• docs
• edits
• fan-in-producer
• feat/add-credo-and-fix-issues
• feat/add-is-full-to-batch-info
• feat/config-storage-behaviour
• feat/telemetry-span
• fix-crash_reason-for-erlang-errors
• fix-telemetry-docs
• fix-typo
• fix-typos
• fix/batcher-fullsweep-after
• fix/ensure-ets-table-exists
• fix_typos
• labels
• main
• master
• patch-1
• ps-add-batcher-key-telemetry-md
• ps-add-more-telemetry-metadata
• ps-add-topology-function
• ps-change-persistent-term-key-value
• ps-fix-intermittent-test
• ps-fix-telemetry-events-and-docs
• ps-topology-fun-from-process
• ps-update-telemetry-docs-2
• refs/pull/328/merge
• refs/pull/329/merge
• sdc/add-producer
• subscriber-remove-shuffle
• switching_to_counters
• telemetry-failed-messages
• wm-ci

Build # 0160e33666b8ae8fdfb245bb0ac3b8d5d0a49817-PR-362

Build Type

Pull #362

github

Committed by rslota

Commit Message

Fix graceful shutdown race condition

Currently, when `Topology` is shutting down, it calls `Terminator.trap_exit/1` to allow it to handle `terminate/2` callback and shutdown producers gracefully.
However, `Terminator.trap_exit/1` is using `GenServer.cast/2` which makes it fully asynchronous, making it possible for `Topology`'s
terminate callback to return before `Terminator` starts trapping exits, allowing `Terminator` to be shut down without putting producers into "draining" state.

In my system when we're running ~50-100 Broadway instances we're seeing ~50% of them shutting down properly and the rest of them get stuck with producers
going full blast unaware of the shutdown.

This change simply changes `Terminator.trap_exit/1` to use `GenServer.call/2` instead of `GenServer.cast/2` to make it fully synchronous, which fixed the issue.
Since I don't know Broadway internals at all, please let me know if there is a better way to fix this.

Pull Request Pull Request #362: Fix graceful shutdown race condition

Run Details

2 of 2 new or added lines in 1 file covered. (100.0%)

2 existing lines in 2 files now uncovered.

636 of 684 relevant lines covered (92.98%)

144.42 hits per line