Logflare / logflare / 83564d927e4ba5feb6b6a86ae0f9f8433c43a18b-PR-3674

81%

main: 81%

Build Type

Pull #3674

github

Committed by djwhitt

Commit Message

feat(telemetry): expose batch flush trigger and per-backend missing IDs

Adds metrics to diagnose small/under-filled ClickHouse inserts (a driver of
TOO_MANY_PARTS): whether batches flush on :size vs :timeout, and how many
event IDs go missing from ETS between claim and batch-insert.

- Move `batch_trigger` from the handle_batch measurements map into metadata
  across all five pipeline emitters (clickhouse, bigquery, webhook, http_based,
  spool) so Telemetry.Metrics can tag by it. `batch_size` stays in measurements;
  the dev dashboard (the only measurements consumer) is unaffected.
- Register `logflare.backends.pipeline.handle_batch.batch_trigger` counter
  tagged by `backend_type` + `batch_trigger`.
- Tag the existing `missing_ids` sum by `backend_id`/`event_type` and add a
  distribution, normalizing heterogeneous emitter metadata (ClickHouse carries
  backend_id+event_type, BigQuery carries source_id) via a tag_values fallback.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Pull Request Pull Request #3674: feat(telemetry): expose batch flush trigger and per-backend missing IDs

Coverage Stats

10 of 11 new or added lines in 6 files covered. (90.91%)

7 existing lines in 3 files now uncovered.

14040 of 17373 relevant lines covered (80.82%)

12419.3 hits per line