Logflare / logflare / 34979239037-1

83%

main: 83%

Build Type

Pull #3991

github

Committed by Baishan

Commit Message

Replace spool Partition/Committer stack with durable_buffer

Rebuilds the spool producer's write path on the durable_buffer library
instead of the hand-rolled Partition/Committer/Buffer design: a new
DurableBuffer.Backends.Cloud backend commits straight to GCS/S3 for
:mem buffer mode, and DurableBuffer.Backends.RotatingWal wraps it with
a local-disk WAL (fsync group-commit, size/time-based rotation to
cloud storage, crash recovery of both sealed and unflushed segments)
for :wal mode, the new default.

Spool.Health tracks write health for the producer-mode gate. The old
ETS+Broadway ProducerPipeline (already dead code from an earlier spool
iteration) and its now-redundant ProducerSup wrapper are removed along
with it. Consumer-side decoding switches from a bespoke Framing module
to durable_buffer's own (bit-compatible) WAL framing. A dedicated
Logflare.FinchSpool connection pool isolates the spool's GCS/PubSub
HTTP clients from the rest of the app's traffic.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Pull Request Pull Request #3991: Replace spool Partition/Committer stack with durable_buffer

Coverage Stats

264 of 289 new or added lines in 16 files covered. (91.35%)

14 existing lines in 3 files now uncovered.

15102 of 18209 relevant lines covered (82.94%)

2215.25 hits per line