vnvo / deltaforge / 23923437008

45%

Build Type

push

github

Committed by web-flow

Commit Message

Operational improvements: labels, checkpoints, structured errors, production dashboard (#67)

* feat: pipeline labels, checkpoint inspection API, per-table lag

Pipeline metadata: labels + annotations (HashMap<String, String>).
Labels enable filtering via GET /pipelines?label=env:prod (AND logic).

Checkpoint inspection: GET /pipelines/{name}/checkpoints returns
per-sink checkpoint positions with age.

Per-table replication lag: deltaforge_source_table_lag_seconds{table}
gauge emitted per table in each batch alongside pipeline-level lag.

Correctness test matrix added to guarantees.md — maps every guarantee
to its test with status (exists/planned).

* feat: per-table replication lag metric (deltaforge_source_table_lag_seconds)

* docs: correctness test matrix in guarantees page

* docs: update observability page — mark implemented metrics, add DLQ + per-table lag

* feat: pipeline labels, annotations, and REST filtering

Metadata: labels (HashMap) and annotations (HashMap) with serde(default).
GET /pipelines?label=env:prod - AND logic, key:value or key-only filter.
Labels enable Grafana variables, operator selection, fleet management.

* feat: enriched pipeline status, health details, checkpoint API

GET /pipelines/{name}: ops field with DLQ count, per-sink checkpoints.
GET /health: returns JSON with status + failed_pipelines list (was plain text).
GET /pipelines/{name}/checkpoints: per-sink positions and ages.
deltaforge_pipeline_info{pipeline,tenant} gauge for Grafana joins.

* feat(grafana): production-scale dashboard for 300+ pipelines

Rebuilt dashboard optimized for fleet operations:
- Fleet overview: aggregate totals (events/s, data/s, max lag, DLQ, errors)
- Top-N panels: top 10 laggiest, throughput, DLQ backlogs (readable at scale)
- Tenant variable for filtering by tenant label
- Per-table lag (top 10 tables)
- All DLQ metrics: entries, events/s, saturation, overflow
- All EOS metrics: checkpoint status, commit rate, txn commits/aborts
- Tab... (continued)

Run Details

57 of 141 new or added lines in 6 files covered. (40.43%)

1 existing line in 1 file now uncovered.

10567 of 23893 relevant lines covered (44.23%)

155.92 hits per line