pomerium / pomerium / 16732330868

52%

Build Type

push

github

Committed by web-flow

Commit Message

core: use iterators for databroker sync and sync latest (#5742)

## Summary
Refactor the databroker storage interface to use an
`iter.Seq2[*databroker.Record,error]` instead of a `RecordStream`. The
original implementation for the storage backend was written before
iterators existed in Go and we can use the standard pattern now.

I also updated the postgres methods to return lists of records
consistently, and used a much smaller batch size. (Previously only a
single changed record was queried at a time) The latest query now uses
`type` and `id` instead of `OFFSET` which should improve performance
when there are a large number of rows.

## Related issues
-
[ENG-2417](https://linear.app/pomerium/issue/ENG-2417/core-use-iterators-for-storage-backends)
-
[ENG-2494](https://linear.app/pomerium/issue/ENG-2494/core-postgres-performance-improvements)

## Checklist
- [x] reference any related issues
- [x] updated unit tests
- [x] add appropriate label (`enhancement`, `bug`, `breaking`,
`dependencies`, `ci`)
- [x] ready for review

Coverage Stats

209 of 313 new or added lines in 8 files covered. (66.77%)

16 existing lines in 7 files now uncovered.

23531 of 45254 relevant lines covered (52.0%)

76.07 hits per line