Logflare / logflare / fde44b4e5d3f0a232831987dc49dbcc64751137b

82%

Build Type

push

github

Committed by web-flow

Commit Message

fix(ingest queue): tolerate dead ETS table when truncating (#3647)

* fix(ingest queue): tolerate dead ETS table when truncating

QueueJanitor crashes were caused by an ArgumentError in the truncate
path when a producer-owned ETS table was reclaimed mid-call. The
guards (get_tid != nil, :ets.info/2 size) only narrow the window: the
owning producer can still die between tid resolution and the
:ets.delete_all_objects / select_delete / insert calls, raising
ArgumentError that terminates the janitor gen_server (~60 crashes/6h
in prod).

Split tid resolution from the ETS work by extracting a public
truncate_tid/3 (the tid-based counterpart to truncate_table/3,
mirroring delete_id/2 and update_status/3). Both clauses rescue
ArgumentError, emit the existing :stale_table telemetry, and return
{:error, :not_initialized} instead of crashing. The size-bounded
clause also tolerates an :ets.info/2 :undefined result (table
reclaimed before the size read) via its with/else. The caller
(QueueJanitor.drop_queue/5) already ignores the return value.

The extracted truncate_tid/3 is also a test seam: a regression test
drives it with a reclaimed tid (the window truncate_table/3 hides
behind get_tid/1's liveness check) and asserts graceful handling plus
telemetry. Without the rescue these tests raise the prod ArgumentError.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* collapse identical telemetry calls into private `emit_stale_ets_table_telemetry/0` func

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Adam Mokan <amokan@gmail.com>

Coverage Stats

12 of 13 new or added lines in 1 file covered. (92.31%)

3 existing lines in 2 files now uncovered.

13558 of 16576 relevant lines covered (81.79%)

5113.78 hits per line