Logflare / logflare / e68633a73a549f4797ff8c21d67e6c0aa4ffbe5d-PR-3616

81%

main: 82%

Build Type

Pull #3616

github

Committed by djwhitt

Commit Message

perf(bq pipeline): emit ingest telemetry in handle_batch, drop ack ETS lookup

ack/3 re-fetched the full LogEvent from ETS per successful event solely to
resolve telemetry labels — a second copy of an event handle_batch/4 had
already pulled out of ETS. Emit the per-event ingest telemetry in
handle_batch instead, where the LogEvent is already in hand via `triples`,
resolving the label mapping and backend metadata once per batch.

ack now only needs each event's id to finalize queue state and never touches
the event body, removing the per-event lookup entirely (both the labelled and
unlabelled paths). Replaces the maybe_emit_event_telemetry machinery added in
the previous commit.

Behavior note: telemetry now fires at batch-insert time rather than at ack.
The set is identical (handle_batch's succeeded == ack's successful) except on
the exceptional path where handle_batch raises after emitting — consistent
with the pipeline already emitting regardless of BigQuery insert errors.
Events missing from ETS are not in `triples`, so they emit no ingest
telemetry (matching the original pre-skip behavior).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Pull Request Pull Request #3616: perf(ingest): update_counter claim + BigQuery pipeline allocation cleanups

Coverage Stats

24 of 26 new or added lines in 4 files covered. (92.31%)

2 existing lines in 1 file now uncovered.

13226 of 16292 relevant lines covered (81.18%)

5083.48 hits per line