FROSADO / onecloudriver / 32005239988

78%

push

github

perf(fs): streaming delta apply with bounded memory (issue #68) (#99)

pollAndApply previously accumulated every delta item into a map across all
pages before applying anything, and only persisted the delta link after the
whole cycle succeeded. Now each page is applied as it arrives and the delta
link is persisted per page:

- Memory stays O(page size): a 100k-item delta cycle peaks ~9 MiB instead of
  holding the full batch in heap (regression-tested, < 50 MiB criterion).
- A crash or transient failure resumes from the last applied page: the delta
  loop now resumes from the persisted per-page link on error, so only the
  failed page is re-fetched on retry instead of the whole cycle.
- Folder deletions blocked by a non-empty directory keep the cross-page
  retry semantics of the previous full-set second pass via a bounded pending
  list, so no ghost folders are left behind.
- Deltas are ordered and idempotent, so per-page sequential application
  converges to the same final state as the previous last-wins dedupe.

Tests: partial-failure (page 1 applied, page 2 fails -> resume completes),
end-to-end delta-loop resume with BoltDB (initial page fetched exactly once),
bounded-memory (100k items, parent not in cache, peak heap < 50 MiB). Both
regression tests fail against the previous map-accumulation implementation.

Closes #68

32 of 43 new or added lines in 1 file covered. (74.42%)

4385 of 5603 relevant lines covered (78.26%)

3150.06 hits per line