Logflare / logflare / 575352326abd8c88cbd94796bdd933f14ced62cb

82%

Build Type

push

github

Committed by web-flow

Commit Message

perf(bq pipeline): allocation cleanups in handle_batch + streaming insert (#3618)

* perf(ingest queue): claim pending events via update_counter CAS-lite

take_pending_ids/2 claimed each event with a per-id :ets.select_replace,
which compiles a match spec per id in the BIF layer — the dominant cost on
the BigQuery claim hot path.

Replace it with an atomic claim counter held in a new 5th tuple field. The
row is now {event_id, status, event, size, claim}, and a claim succeeds when
:ets.update_counter/3 transitions claim from 0 to 1. This is multi-consumer
safe (only the 0->1 winner takes the event) like select_replace, but uses a
direct integer op with no per-id match-spec compilation, and needs no dedup
pass since write_concurrency resize-duplicates lose the CAS naturally.

Re-insertion resets the counter to 0, so both requeue paths keep events
claimable: BufferProducer failure requeue (via add_to_table) and the
QueueJanitor stale :processing -> :pending reset. Also drops a redundant
:ets.info(:size) read in take_pending_ids, since get_tid/1 already confirms
the table is live.

Benchmarks show ~4.9x faster claims (652 -> 132 us per 1000-event batch).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(ingest queue): take_pending_ids tests and claim-strategy benchmark

Add a take_pending_ids/2 describe block covering claim correctness, the
remainder-left-pending case, no re-claim across sequential calls, and a
concurrent-claim race test asserting no event is claimed twice (the
multi-consumer guarantee).

Add a profiling benchmark comparing claim strategies (update_counter,
select_replace, update_element, single compiled select_replace) with
recorded results and notes on why the single-compile approach is rejected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(ingest queue): cover claim-reset reclaim cycle for take_pending_ids

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
... (continued)

Coverage Stats

7 of 7 new or added lines in 1 file covered. (100.0%)

1 existing line in 1 file now uncovered.

13212 of 16279 relevant lines covered (81.16%)

5265.53 hits per line