Logflare / logflare / 33779696421-1

82%

main: 83%

Build Type

Pull #3905

github

Committed by chasers

Commit Message

fix: keep database work out of the SourceSup partition critical section

Two canary nodes wedged at boot, each holding ~52 GiB of process heap. ErlSysMon
named a SourcesSup partition with a long_message_queue on both, 14 seconds before
either node appeared in telemetry.

SourcesSup is a PartitionSupervisor of DynamicSupervisors, and sources hash to a
partition by id. `DynamicSupervisor.start_child/2` is a `GenServer.call` with an
infinite timeout, and `SourceSup.init/1` runs inside it. That init does five
cache-fronted lookups, each of which falls through to Postgres on a miss. A cold
node therefore makes several database round trips per source start, all
serialized through one partition, which blocks every other source hashing to it.

`SourceSup.prefetch/1` resolves exactly those five lookups, and
`start_source_sup/1` calls it before `start_child`. The latency moves to the
calling process, where it parallelizes across callers, so the partition's
critical section only ever reads warm Cachex entries.

`init/1` is unchanged, so a crash-restart still re-resolves and picks up
configuration changes.

Rejected alternative: passing a pre-resolved struct into the child spec and making
`init/1` pure. `restart: :transient` means DynamicSupervisor reuses the stored spec
on a crash-restart, so the supervisor would come back with a stale plan and backend
list indefinitely.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XJWSbFFxwiYB25iUAGmXc

Pull Request Pull Request #3905: fix: keep database work out of the SourceSup partition critical section

Coverage Stats

6 of 6 new or added lines in 2 files covered. (100.0%)

5 existing lines in 2 files now uncovered.

14722 of 17912 relevant lines covered (82.19%)

2225.8 hits per line