Logflare / logflare / 875092b0840ed6342af0a2570c4dfeea7f07894b-PR-3635

82%

main: 82%

Build Type

Pull #3635

github

Committed by djwhitt

Commit Message

fix(ingest queue): tolerate dead ETS table on add_to_table insert

A BufferProducer owns its per-queue :ets table (no heir), so the table is
reclaimed the moment the producer process exits — routine during
DynamicPipeline scale-down. add_to_table/3 resolves a tid via get_tid
(which liveness-checks the table) and then inserts, leaving a TOCTOU
window: the producer can die between the check and the :ets.insert,
raising ArgumentError that propagates up through dispatch_to_backends and
crashes the whole ingest batch.

Wrap the insert in a try/rescue. On ArgumentError, re-route the batch to
the supervisor-owned startup queue ({sid, bid, nil} /
{:consolidated, bid, nil}) — the same queue a clean producer exit drains
to, and which the scaler treats as a scale-up signal so a new producer
picks it up. Bottom out with {:error, :not_initialized} if the startup
queue is gone too, guarding against recursion. Emit the existing
:stale_table telemetry so the fallback is observable.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Pull Request Pull Request #3635: fix(ingest queue): tolerate dead ETS table on add_to_table insert

Coverage Stats

6 of 6 new or added lines in 1 file covered. (100.0%)

2 existing lines in 2 files now uncovered.

13422 of 16449 relevant lines covered (81.6%)

5234.64 hits per line