Unleash / unleash / 32377185107

87%

master: 91%

Build Type

push

github

Committed by web-flow

Commit Message

feat(metrics): add token cache watermark

### Issue

Our plan for the cache validation on API tokens is to introduce initial
metrics that check in production before we build on it.
This PR is [on top
of](https://github.com/bricks-software/unleash-enterprise/pull/1529)
that introduces some of these metrics.

Maybe the most important metric is to validate that every token change
_actually_ writes an event. So, we can drop the fetchActiveTokens() (60'
refresh) and drive invalidation off the events table instead.

We want to introduce this metric on this PR.

### The idea

We add an event watermark for tokens, polled by max(id) of the
api-token- events. We reload only when this moves.

And after the 60'' refresh, we already have, we can check: 
-"did the _token_ set change?"
-and "did the _event watermark_ move?"
If the two disagree, it means something changes tokens without an event,
and if a cache trusting events would give out old tokens.
When only the watermark has moved, we mark it as a
token_revision_advance_total counter.
If only the set has changed, as a missed_changed counter.

Before, we build anything on top, we need to make sure this
"missed_changed" is zero.
If we've many "advanced" counters, we better know it now than laters :p
(might be a mark to redesign and avoid full reloads?)

We already suspect two paths that currently can give us cache misses:

- the `deleteSystemCreatedTokensNotSeen` calls the store directly,
deletes tokens of Edge, but does not emit any storeEvent
- v2's env ... the ON DELETE CASCADE deletes tokens, but nothing in our
code observes it

we might miss others too. But, now we can find them out ;)

### How we do this

- Add new metric event `token_revision_probe`
- added `getMaxTokenRevisionId`() on the event store. Separate from
getMaxRevisionId on purpose, token changes must not advance the
configuration revision or they'd invalidate feature ETags
- scheduleServices adapts to take stores as a third arg so the probe ca... (continued)

Coverage Stats

3858 of 5021 branches covered (76.84%)

0 of 6 new or added lines in 1 file covered. (0.0%)

3 existing lines in 1 file now uncovered.

15904 of 18187 relevant lines covered (87.45%)

791.87 hits per line