Logflare / logflare / d0bdf1c243a4bb2767be56cb30dcdb05ced5d9e9

82%

Build Type

push

github

Committed by web-flow

Commit Message

fix(ingest queue): tolerate dead ETS table on add_to_table insert (#3635)

A BufferProducer owns its per-queue :ets table (no heir), so the table is
reclaimed the moment the producer process exits — routine during
DynamicPipeline scale-down. add_to_table/3 resolves a tid via get_tid
(which liveness-checks the table) and then inserts, leaving a TOCTOU
window: the producer can die between the check and the :ets.insert,
raising ArgumentError that propagates up through dispatch_to_backends and
crashes the whole ingest batch.

Wrap the insert in a try/rescue. On ArgumentError, re-route the batch to
the supervisor-owned startup queue ({sid, bid, nil} /
{:consolidated, bid, nil}) — the same queue a clean producer exit drains
to, and which the scaler treats as a scale-up signal so a new producer
picks it up. Bottom out with {:error, :not_initialized} if the startup
queue is gone too, guarding against recursion. Emit the existing
:stale_table telemetry so the fallback is observable.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Coverage Stats

6 of 6 new or added lines in 1 file covered. (100.0%)

1 existing line in 1 file now uncovered.

13437 of 16461 relevant lines covered (81.63%)

5220.39 hits per line