vnvo / deltaforge / 23717982936

45%

Build Type

push

github

Committed by web-flow

Commit Message

Exactly-once delivery with per-sink independent checkpoints (#62)

* feat: exactly-once delivery with per-sink independent checkpoints

Kafka transactional producer (begin/commit per batch), NATS Nats-Msg-Id
server-side dedup, Redis idempotency_key for consumer-side dedup.

Per-sink checkpoints: each sink advances independently, source replays
from min(sink checkpoints). Legacy checkpoint fallback for seamless
migration. Source::compare_checkpoints trait method for correct ordering
(MySQL file+pos, Postgres LSN, Turso change_id).

Coordinator: policy gate before checkpoint commits, parallel per-sink
commits via join_all, SinkError::Fatal for producer fencing.

Benchmarks (tuned, single dev machine):
- MySQL:    151K events/s (at-least-once) / 134K (exactly-once) — 11% overhead
- Postgres:  57K events/s (at-least-once) /  53K (exactly-once) —  7% overhead
- Default max_bytes bumped 3MB → 16MB to prevent batch byte-capping

Chaos: exactly-once crash recovery scenario, configurable drain target/
writers/timeout/max_bytes, effective config dump before each run.

Infra: Docker Compose profiles (base/mysql-infra/pg-infra/kafka-infra),
Loki+Promtail log aggregation, Grafana dashboard with template variables
(instance/pipeline/source/sink/processor), per-sink checkpoint and
transaction commit/abort panels.

Docs: updated performance guide, Kafka/NATS/Redis sink references,
per-sink checkpoint architecture, delivery guarantee tiers.

* fix: flush partial batch when source goes idle at WAL/binlog tail

recv_many blocked indefinitely when no events were available, preventing
the select! loop's ticker branch from flushing the partial batch. Wrap
recv_many in tokio::time::timeout(tick_ms) so control returns to the
timer when the source goes idle.

Adds regression test: test_partial_batch_flushed_by_timer sends 3 events
to a coordinator with max_events=10000, keeps the channel open, and
asserts the timer flushes the partial batch within max_ms.

* fix: clippy lints... (continued)

Run Details

322 of 711 new or added lines in 16 files covered. (45.29%)

7 existing lines in 5 files now uncovered.

9819 of 22661 relevant lines covered (43.33%)

164.0 hits per line