pomerium / pomerium / 17307291904

52%

Build Type

push

github

Committed by web-flow

Commit Message

core: on-disk databroker storage (#5774)

## Summary
This adds a new `file` databroker storage provider that stores data
on-disk. The DSN format is: `file:///path/to/file`.

This storage provider is implemented with a pebble key-value store.
Pebble implements a global, sorted collection of key-value pairs. We
break up the databroker storage into various "key spaces". Each key
space is distinguished via a single-byte prefix. All the keys for that
key space will have the same prefix.

- `Lease`: used to implement distributed locking: key is the lease name,
value is the lease id and the expires at timestamp.
- `Metadata`: metadata about the store: server version and migration
version.
  - The server version is randomly generated for an empty database. 
  - Migrations are used for future changes to the on-disk format.
- `Options`: options for a record type: key is the record type, value is
the protobuf-encoded options.
- Options are currently only used to set a capacity limit for a record
type (for example for events).
- `RecordChange`: each record as they are updated: key is the version,
value is the protobuf-encoded record.
- Record changes are like a write-ahead log. They are used by clients
via the `Sync` call to receive changes as they are made.
- In addition to `Put`s, `Delete`s are also stored as record changes.
(tombstones)
- `RecordChangeIndexByType`: an index of record type to version: key is
the record type + the version, value is nil.
- This index allows us to implement `Sync` with a record type filter
efficiently.
- `Record`: the latest version of each record: key is the record type +
the record id, value is the protobuf-encoded record.
- `RecordIndexByTypeVersion`: an index of record type and version to
record id: key is the record type + the version, value is the record id.
  - This index is used to enforce capacity limits.
- `RegistryService`: registered services: key is kind + endpoint, value
is the expires at timestamp.
- We maintain an ... (continued)

Coverage Stats

1138 of 1708 new or added lines in 18 files covered. (66.63%)

26 existing lines in 7 files now uncovered.

25582 of 47461 relevant lines covered (53.9%)

90.2 hits per line