zalando / skipper / 26750428017

80%

Build Type

push

github

Committed by web-flow

Commit Message

introduce timeout for decicion log task (#4035)

## Problem

When asyncDecisionLogging is enabled and a new policy bundle is
deployed, eopa_dl.Reconfigure() calls Stop() on the running Benthos
stream followed by Start() on a new one. If `runDecisionLogger` is
inside stream.Consume() on the old stream at that moment, the Benthos
producer closure tries to write the decision event to an internal
channel whose reader (inproc input) has been destroyed by Stop().
  
Because `doLogDecision` was called with context.Background() (without
deadline) the blocked send never returns. `runDecisionLogger` is
permanently stuck on that one call and stops draining `decisionLogChan`.

With the goroutine stuck, once the channel fills to capacity
(buffer_size_limit_events), every subsequent `logDecision` call hits the
non-blocking default branch and drops the event, logging "Decision log
dropped: async buffer full." The warning continues indefinitely with no
self-recovery. The only fix prior to this change was a pod restart.
 
## Fix
Set timeout for the context passed for decision logging. The value for
the timeout is decided using the configured timeout for the eopa_dl
output type

---------

Signed-off-by: nuwandi-wickramasinghe_zse <nuwandi.wickramasinghe@zalando.de>
Co-authored-by: nuwandi-wickramasinghe_zse <nuwandi.wickramasinghe@zalando.de>

Coverage Stats

24 of 44 new or added lines in 2 files covered. (54.55%)

13 existing lines in 4 files now uncovered.

28074 of 35021 relevant lines covered (80.16%)

83387.67 hits per line