pomerium / pomerium / 16725584523

52%

Build Type

push

github

Committed by web-flow

Commit Message

databroker: move gc to public interface, abort sync calls with invalid record versions (#5753)

## Summary
The databroker maintains a list of record changes over time. Those
changes are retrieved via the databroker `Sync` call. Both the in-memory
and postgres storage backends periodically remove old changes.

Since this logic is necessary for any storage backend, add a `Clean`
method to the public interface and start the background cleaner in the
databroker server rather than the storage backend itself. This makes it
easier to test.

Currently if a client attempts to `Sync` changes, but they've fallen too
far behind, we would not detect this and the client would never receive
the missing changes. The correct behavior is to abort the `Sync`, which
signals the client to do a full re-sync via `SyncLatest`. This PR
attempts to implement that behavior.

There is some risk with this change. If the assumptions I'm making about
how clients work turn out not to be true, we could end up with clients
continuously attempting to call `Sync` and never recovering.

## Related issues
-
[ENG-2640](https://linear.app/pomerium/issue/ENG-2640/core-databroker-sync-may-miss-changes)


## Checklist

- [x] reference any related issues
- [x] updated unit tests
- [x] add appropriate label (`enhancement`, `bug`, `breaking`,
`dependencies`, `ci`)
- [x] ready for review

Coverage Stats

71 of 143 new or added lines in 6 files covered. (49.65%)

16 existing lines in 6 files now uncovered.

23664 of 45444 relevant lines covered (52.07%)

75.71 hits per line