pomerium / pomerium / 17924876307

52%

Build Type

push

github

Committed by web-flow

Commit Message

feat: report pomerium health to systemd (#5835)

## Summary

This PR adds systemd as a valid integration for internal pomerium health
checks.

When all expected checks have started, report READY to systemd. 

When any component enters the terminating state, report STOPPING to
systemd.

When any component reports an error, the latest error reported gets
surfaced to systemd's status so that `sudo systemctl status
pomerium.service` reflects the last error

References (sd_notify & sd_watchdog_enabled man pages), convenient web
links here:
- https://www.freedesktop.org/software/systemd/man/latest/sd_notify.html
-
https://www.freedesktop.org/software/systemd/man/latest/sd_watchdog_enabled.html#

To have a startup probe based on the sd_notify protocol the systemd
configuration should have:
```
[Service]
#...
Type=notify
```

The sd_notify health checker also implements the watchdog part of the
systemd configuration:

```
[Service]
# ....
WatchdogSec=30s
```

Currently watchdog heartbeats regularly trivially as long as the process
(and sd_notify goroutine) is up and running.

This could be extended in the future to have the equivalent of a
liveness probe via the watchdog messages, where the sd_notify health
checker will no longer send watchdog messages if it is unhealthy.


## Related issues


[ENG-2902](https://linear.app/pomerium/issue/ENG-2902/systemd-sd-notify-health-check-implementation)


## User Explanation

Users installing pomerium via systemd now benefit from pomerium's
internal health checks being integrated with `systemd` via `sd_notify`.

## Checklist

- [x] reference any related issues
- [x] updated unit tests
- [x] add appropriate label (`enhancement`, `bug`, `breaking`,
`dependencies`, `ci`)
- [x] ready for review

Coverage Stats

142 of 199 new or added lines in 3 files covered. (71.36%)

7 existing lines in 3 files now uncovered.

27184 of 50436 relevant lines covered (53.9%)

86.79 hits per line