pomerium / pomerium / 17474795591

52%

Build Type

push

github

Committed by web-flow

Commit Message

feat: handle signals to better gracefully drain/terminate envoy (#5818)

## Summary

Right now we explicitly do not handle OS signals in pomerium - this has
the side effect of not necessarily running shutdown phases of pomerium,
e.g. things that are supposed to run after a context is done, for
example `context.AfterFunc(ctx, ...)`.

It works mostly as expected right now since when container orchestration
frameworks like Kubernetes pass a SIGTERM to the process, any
sub-processes spawned with exec.Command will inherit those signals and
get sent SIGTERM as well, which according to the envoy docs will result
in draining listeners according to `--drain-time` which defaults to
600s.

However, right now the pomerium.Shutdown is a utility function
explicitly used by our test environment, our current pomerium.Wait
doesn't handle graceful shutdown correctly, it has been amended to:
- wait for envoy server to close (either successfully or unsuccessfully)
- return errors to the parent command if and only if the first error was
not surfaced via pomerium.Shutdown() or by one of the valid interrupt
signals. Not handling these errors means that systemd, will handle
graceful shutdown as well since it expects a zero return code on
successful shutdown, which the previous iteration did not have.


An unintended side effect of catching signals is the envoy server
gracePeriodSeconds on is always set to 0 (in non-test env settings) --
causing the envoy process to be SIGKILLED inadvertently by the old code.
This has been fixed by respecting the default `--drain-time`.

<hr/>
In addition, we make a best effort to post an explicit graceful drain
request to envoy during shutdown.

<hr/>

Some testing utilities I used to verify different termination sceanrios:
- `strace -p $(pidof pomerium) -e signal`
- `strace -p $(pidof envoy) -e signal`

should always receive SIGTERM/SIGINT and exit with status code 0. 

<hr/>

Ref:
-
https://www.envoyproxy.io/docs/envoy/latest/intro/arch_ov... (continued)

Coverage Stats

0 of 85 new or added lines in 5 files covered. (0.0%)

19 existing lines in 7 files now uncovered.

25952 of 47991 relevant lines covered (54.08%)

89.83 hits per line