Logflare / logflare / e38c122acdf43657a1fa3066273b93f723539cf9

82%

Build Type

push

github

Committed by web-flow

Commit Message

perf(bq pipeline): emit ingest telemetry in handle_batch, drop ack ETS lookup (#3619)

* perf(bq pipeline): skip per-event ETS lookup in ack when source has no labels

ack/3 looked up the full LogEvent from ETS for every successful event solely
to resolve telemetry labels, copying the entire event out of the cache table
per event. The label mapping was also re-parsed from source.labels on every
event (get_labels_from_event called get_labels_mapping per call).

Resolve the label mapping once per ack. When it is empty — the common case
for sources without configured labels — the event body is never read, so skip
the ETS lookup entirely and emit telemetry directly. When labels are present,
the lookup is still required and unchanged.

Adds Sources.extract_labels/2 (mapping-aware extraction) with get_labels_from_event/2
delegating to it. Behavior note: a successful event missing from ETS at ack
time now still emits ingest telemetry in the no-labels case (previously the
lookup miss suppressed it) — correct, since the event was confirmed inserted.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(bq pipeline): emit ingest telemetry in handle_batch, drop ack ETS lookup

ack/3 re-fetched the full LogEvent from ETS per successful event solely to
resolve telemetry labels — a second copy of an event handle_batch/4 had
already pulled out of ETS. Emit the per-event ingest telemetry in
handle_batch instead, where the LogEvent is already in hand via `triples`,
resolving the label mapping and backend metadata once per batch.

ack now only needs each event's id to finalize queue state and never touches
the event body, removing the per-event lookup entirely (both the labelled and
unlabelled paths). Replaces the maybe_emit_event_telemetry machinery added in
the previous commit.

Behavior note: telemetry now fires at batch-insert time rather than at ack.
The set is identical (handle_batch's succeeded == ack's successful) except on
the exception... (continued)

Coverage Stats

12 of 14 new or added lines in 2 files covered. (85.71%)

6 existing lines in 3 files now uncovered.

13413 of 16444 relevant lines covered (81.57%)

5027.61 hits per line