codenotary / immudb / 33902119093

85%

master: 85%

Build Type

Pull #2157

gh-ci

Committed by vchaindz

Commit Message

feat(server): report databases that failed to open on /readyz

Health returns Status: true unconditionally and DatabaseHealth reports request
statistics, so neither can tell an orchestrator that a database is unusable.
DatabaseListV2 is not an answer either: Loaded is !IsClosed(), and closed is
set only by Close() or PutClosed(), never by a failed open, so a database
whose open failed still reports Loaded: true.

The consequence is that when a transient 403 from S3 left two databases
unopened (#2155), the pod stayed 1/1 Running and the liveness probe passed for
the whole outage. It surfaced instead as a crashloop in the client services,
pointing on-call at the consumers rather than at immudb.

The metrics server already serves /initz, /readyz and /livez, all of them
unconditional 200 stubs. /readyz now returns 503 and names each database whose
most recent open attempt failed and which has not opened successfully since.
/livez and /initz stay unconditional: liveness should not depend on storage
reachability.

A database that was never accessed has never attempted an open and is not
reported, so registering databases lazily at startup does not fail readiness.
The failure record is read atomically rather than under the dbInfo mutex,
which is held for the entire duration of an open and so is precisely
unavailable when a probe needs an answer.

IsActive is corrected in passing: it only checked that the cache lookup
succeeded, which was true for a ref whose open had failed. It now requires a
ref that actually holds an open database.

Closes #2156

Signed-off-by: vchaindz <dennis@codenotary.com>

Pull Request Pull Request #2157: fix(database): make a failed database open recoverable, and report it on /readyz

Coverage Stats

110 of 120 new or added lines in 6 files covered. (91.67%)

5 existing lines in 3 files now uncovered.

45357 of 53449 relevant lines covered (84.86%)

122000.49 hits per line