codenotary / immudb / 25122424304

85%

Build Type

push

gh-ci

Committed by vchaindz

Commit Message

fix(store): grow precommit buffer during recovery instead of failing open

Issue #2086: an ungraceful shutdown (e.g. Kubernetes OOMKill) that left
the txLog ahead of the cLog by more than MaxActiveTransactions used to
make the store permanently unopenable. Reopen failed with

    ERROR: unable to open database: buffer is full: while loading
           pre-committed transaction: 2367

and the database was stuck restart-looping until the operator either
bumped MaxActiveTransactions past the actual on-disk backlog or
hand-trimmed the txLog. The S3-backed reporter hit it after their pod
was OOMKilled mid-async-flush; the bug is not S3-specific.

Root cause
----------
embedded/store/immustore.go:556 sizes the recovery buffer at
MaxActiveTransactions (default 1000). That cap is correct for the
runtime role of cLogBuf (it back-pressures new precommit calls so the
in-flight pipeline stays bounded), but wrong for the recovery role:
the loader at immustore.go:568-595 reads every pre-committed-but-not-
cLog-committed tx from the txLog into that fixed-size circular buffer,
and the on-disk backlog after a crash can exceed the runtime cap by
arbitrary amounts. The (count - MaxActiveTransactions + 1)-th put
returned ErrBufferIsFull and the loader bailed.

Fix
---
- precommit_buffer.go: add grow(newSize) that expands the buffer
  in place, preserving occupied entries in their logical (insertion)
  order. Concurrent access is not supported; only the single-threaded
  recovery path calls it.
- immustore.go: when cLogBuf.put returns ErrBufferIsFull during the
  recovery loop, grow the buffer to 2x and retry. Doubling keeps
  allocation amortised O(N) and bounds peak usage at <2x the backlog;
  each slot is a 48-byte precommittedEntry, so even a 100k-tx backlog
  is ~5 MB - negligible against vLogCache and AHT.

After recovery completes the buffer stays at its grown size for the
lifetime of the store. That memory is the price of being able to
reopen at all.

Tests
-----
... (continued)

Coverage Stats

31 of 31 new or added lines in 2 files covered. (100.0%)

2 existing lines in 1 file now uncovered.

44644 of 52365 relevant lines covered (85.26%)

127357.49 hits per line