Logflare / logflare / 6813ce93aaa8c19356ac35724ba805bc5b491632

81%

Build Type

push

github

Committed by web-flow

Commit Message

perf: reduce process heap allocations for clickhouse pipeline (#3667)

* Optimize clickhouse pipeline memory using id passing and incremental compression

* Small refactor

* Fix typings

* Fix test

* Fix limit on processing reset

* Prevent test factory creating many source atoms.)

* Format

* Fix issues from merge

* fix: eliminate ClickHouse pipeline CPU regression (#3668)

* fix: eliminate ClickHouse pipeline CPU regression

* fix: correct batcher/batch_key assertion in ClickHouse pipeline transform test

Broadway.Message defaults batcher and batch_key to :default (not nil), so the
"unrouted" message produced by transform/2 has those defaults until
handle_message/3 routes it. Assert :default instead of nil.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test: cover metadata fast-path stale routing and take_pending_ids_with_metadata edge cases

Adds coverage for the id_passing_metadata fast path introduced in this PR:
- handle_message/3 routes stale metadata messages to :ch_stale without an
  ETS lookup (previously only the :fresh path was covered)
- take_pending_ids_with_metadata/2 with count 0, an uninitialized table, and
  rows with nil routing fields (the fallback trigger)

Also simplifies the transform/2 fallback to forward the real size instead of a
fabricated 0, since the target clause ignores it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: skip empty ClickHouse insert when a routed batch has no surviving rows

The metadata fast path routes messages in handle_message/3 without re-checking
ETS, so if every row in a batch is deleted (e.g. by the QueueJanitor) before
encode_message/7 runs, stream_compress/4 returns good == [] and a gzip payload
with zero RowBinary rows. finalize_insert/6 previously POSTed that empty payload
to ClickHouse; insert_log_events_compressed/4 has no empty guard (unlike the
non-compressed Ingester.insert/5). Add a good == [] clause that fails the missing
mes... (continued)

Coverage Stats

105 of 124 new or added lines in 5 files covered. (84.68%)

4 existing lines in 3 files now uncovered.

14039 of 17364 relevant lines covered (80.85%)

12446.83 hits per line