zalando / skipper

80%

Build Type

push

github

Committed by web-flow

Commit Message

Add `decision_log.dropped` counter for async decision logging (#4007)

## What & Why?

Follow-up to #4001, which introduced a bounded async channel in front of
`logDecision` so that slow log plugins (e.g. `eopa_dl` writing to S3)
don't add latency to HTTP responses. When the channel is full, events
are silently dropped.

Currently, those drops are only visible as warn log lines. This PR
surfaces the same event as a Prometheus counter so we can:

- **Alert** when drops exceed a threshold (should we decide on one based
on operational experience)
- **Tune `buffer_size_limit_events`** based on observed drop rate.

The counter is emitted via skipper's standard `metrics.Default`:

```
skipper_custom_total{key="decision_log.dropped.<bundle>"}
```

to match the existing OPA counter convention (`decision.allow.<bundle>`,
`decision.deny.<bundle>`, `decision.err.<bundle>`).

_Given naming is hard, please feel free to suggest a better name for the
metric._

Signed-off-by: Santhosh Kannan <29794224+x418@users.noreply.github.com>

Coverage Stats

1 of 1 new or added line in 1 file covered. (100.0%)

9 existing lines in 4 files now uncovered.

27047 of 33815 relevant lines covered (79.99%)

87967.09 hits per line