wegamekinglc / calc-flow / 32999314484

94%

Build Type

push

github

Committed by web-flow

Commit Message

perf: make join epoch capture proportional to dirty state (DAL-160) (#197)

* perf: make epoch checkpoint capture cost proportional to dirty state

The frozen AC19/AC20 baseline (DAL-130, main@f74f32a) quantified two
architecture-level capture violations against the FR47 independence NFR:
at equal dirty cardinality, capture cost grew linearly with total
retained state (8.8 ms @17.5k to 201.7 ms @242.5k rows), and at large
state it blew up superlinearly (45-78.5 s per epoch at ~1.04M retained
rows), so epoch checkpoint timeouts would kill big-state jobs long
before any state-byte limit.

Three layers were each re-reading or re-encoding the full retained
payload every epoch:

- the Join dirty log encoded each upsert by scanning live state for its
  row, O(dirty x state) per capture; upserts now carry the admitted
  record and charge, so delta encoding is O(dirty) with no state scan;
- OperatorStateSnapshot.segments moved owned Vec<u8>, so every carried
  base/delta segment was deep-copied per epoch; the new public
  StateSegment shares one Arc allocation and computes its SHA-256 once,
  and the Join checkpoint now clones cheap handles;
- the manifest transaction re-hashed, re-wrote, and re-validated every
  segment at every epoch, and publish/retention re-read every referenced
  segment; a session-scoped record of committed handles now reuses the
  already-committed handle for unchanged content and skips the
  redundant same-session re-read, while recovery-time selection still
  revalidates every referenced segment byte on every fresh session
  (AC-08). Manifests may reference handles committed at earlier epochs,
  so recovery loads validate handle ownership without a shared epoch.

The tombstone coalescing path in the Join watermark eviction used the
same quadratic pending-log scan and is now a single indexed pass.

A resident stream_join_perf Criterion suite preserves the frozen AC19
baseline scenarios plus a capture-independence canary pair (equal di... (continued)

Coverage Stats

449 of 463 new or added lines in 9 files covered. (96.98%)

7 existing lines in 2 files now uncovered.

65688 of 71901 relevant lines covered (91.36%)

2226.92 hits per line