vnvo / deltaforge / 23816277104

45%

Build Type

push

github

Committed by web-flow

Commit Message

Dead letter queue with per-event failure routing (#63)

* feat: dead letter queue — per-event failure routing

DLQ routes poison events (serialization/routing failures) to a durable
queue backed by StorageBackend, allowing the pipeline to continue with
remaining events.

Core changes:
- JournalEntry + DlqMeta types with payload truncation
- DlqWriter on StorageBackend.queue_* with overflow policies
  (drop_oldest, reject, block with async Notify)
- Sink trait: send_batch returns BatchResult with per-event dlq_failures
- All sinks (Kafka, NATS, Redis, Filter) updated to collect per-event
  serialization/routing errors instead of failing the batch
- Coordinator routes BatchResult.dlq_failures to DlqWriter
- SinkError::is_dlq_eligible() for error classification
- JournalConfig + DlqStreamConfig + OverflowPolicy in pipeline config
- Metrics: dlq_events_total, dlq_entries, dlq_evicted/rejected_total,
  dlq_saturation_ratio, dlq_write_failures_total

Tests (12 new):
- DlqWriter: write/peek, ack, purge, drop_oldest, reject, block
- Coordinator: partial failures → DLQ + continue, all-fail, no-writer
- JournalEntry: truncation, DlqMeta roundtrip

* format and clippy fixes

* feat(dlq): REST API endpoints + PipelineManager wiring

REST: GET /pipelines/{name}/journal/dlq (peek with sink_id/error_kind
filters), GET .../count, POST .../ack, DELETE (purge).

PipelineManager creates DlqWriter when journal.enabled=true and passes
it to the coordinator builder. PipelineController trait extended with
dlq_peek/count/ack/purge default methods.

Fix: RwLock guard not held across await (clone Arc<DlqWriter> first).
Fix: bench pipeline_e2e mock sinks updated for BatchResult return type.

* formatting fixes

* docs: dead letter queue documentation

New DLQ page covering configuration, REST API with examples, overflow
policies, payload truncation, metrics, and limitations.

Updated guarantees page (DLQ no longer a limitation), configuration
reference (journal field), SUMMARY,... (continued)

Run Details

696 of 1130 new or added lines in 29 files covered. (61.59%)

15 existing lines in 14 files now uncovered.

10521 of 23769 relevant lines covered (44.26%)

156.73 hits per line