supabase / etl / 30437075367

74%

Build Type

push

github

Committed by web-flow

Commit Message

Add durable write duration metric to apply loop (#940)

* Add durable write duration metric to apply loop

Add the etl_batch_items_durable_duration_seconds histogram. It
measures the time from batch dispatch until the destination confirms
the batch durable.

The existing send-duration metric records the time until the write
result completes. Deferred destinations complete with Accepted before
the data is durable. Synchronous destinations complete with Durable.
The two groups report different quantities, so a per-destination
latency comparison of that metric is biased.

The apply loop now stores the dispatch instant of each accepted
write. A later durable result covers all earlier accepted writes in
the same ordered stream. The loop records one sample per covered
write with confirmation="deferred", and one sample for the durable
write itself with confirmation="direct".

Empty durability barriers are not recorded. The metric covers
streaming writes only. Table-copy writes settle at a table-wide
barrier and are out of scope here.

* Correct pending durable dispatch queue doc

The write_events contract does not require destinations to bound
their accepted-but-not-durable work, so the old comment overstated
the guarantee. State the real throttle instead: the loop keeps at
most one write in flight, so the queue grows by at most one entry
per completed write and drains at the next durable result.

* Add durable wait duration metric for deferred writes

Add the etl_batch_items_durable_wait_duration_seconds histogram. It
measures the accept-to-durable segment for deferred streaming writes:
the time from the apply loop processing an accepted result until a
later durable result covers that write.

The send and durable duration histograms share the dispatch instant,
so the segment was only derivable by subtracting aggregate means.
Histogram percentiles cannot be subtracted, and the two populations
do not pair one-to-one. Record the segment directly instead: keep ... (continued)

Coverage Stats

58 of 60 new or added lines in 3 files covered. (96.67%)

23 existing lines in 4 files now uncovered.

44269 of 59743 relevant lines covered (74.1%)

8669.77 hits per line