Logflare / logflare / 37055c38e445af7b3f7ca618180422f6c1cf23b6

81%

Build Type

push

github

Committed by web-flow

Commit Message

Feat/s3 producer pipeline (#3662)


- Add a spool-based log ingestion pipeline: producer side buffers/uploads batches to S3/GCS, consumer side downloads and replays them back through the normal ingest path
- Add pluggable queue (SQS, GCP Pub/Sub) and storage (S3, GCS) adaptors behind `Queue`/`Storage` behaviours for the spool pipeline
- Add `enable_spooling` flag on sources (migration + schema + edit-source UI toggle) to opt sources into the spool path
- Add `MemoryMonitor` GenServer for spool backpressure: tracks system memory pressure and per-source destination-queue backlog, throttling the consumer instead of dropping events
- Extend `IngestEventQueue`/`BufferProducer`/`QueueJanitor` with stale `:processing` cleanup and startup-queue migration for spool producers
- Add telemetry throughout the spool pipeline (throttling, queue depth, stale-event handling)
- Add a dev-only LiveView dashboard (`DevDashboard`) for inspecting spool/ingest state locally
- Add GCS/staging provisioning (Cloudbuild config, lifecycle policy) and local docker-compose setups for MinIO/GCP emulation
- Add `scripts/replay_spool_files.exs` for manually replaying spooled files

Coverage Stats

403 of 690 new or added lines in 21 files covered. (58.41%)

1 existing line in 1 file now uncovered.

13981 of 17288 relevant lines covered (80.87%)

6686.1 hits per line