Logflare / logflare / fdc5bff0f9a3067aecdcb65a12d2fbf33531271d-PR-3582

81%

main: 81%

Build Type

Pull #3582

github

Committed by djwhitt

Commit Message

perf: reduce user-metrics puller memory spikes

The user-metrics pull pipeline buffers metrics in an ETS generation and
drains a whole sealed generation into the Broadway pipeline in one step.
With an ~8-10 min export_period and max_demand tied to batch_size, each
drain materializes a large batch of ETS rows into Elixir maps at once,
producing a sharp spike in process memory that scales with metrics volume.

Reshape the drain into a smaller, more frequent, GC-friendly flow:

- Shorten export_period from ~8-10 min to ~30-45s so each generation (and
  thus each drain) is far smaller, trimming runs more often, and the
  post-boot accumulation window before the first drain shrinks accordingly.
- Decouple processor max_demand from batch_size and set it to 250, directly
  bounding the number of rows materialized per drain step.
- Add hibernate_after and spawn_opt fullsweep_after to the Broadway pipeline
  (matching the other ingest pipelines) so the idle producer and processors
  reclaim transient heap and refc binaries between drains.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Pull Request Pull Request #3582: perf: reduce user-metrics puller memory spikes

Coverage Stats

2 of 3 new or added lines in 2 files covered. (66.67%)

2 existing lines in 1 file now uncovered.

12996 of 16048 relevant lines covered (80.98%)

3765.35 hits per line