healthchecks / healthchecks / 23339508851

91%

push

github

Fix sendalerts to avoid crashing on a timed out db connection

MySQL/MariaDB closes idle db connections after a period of inactivity
(controlled by the wait_timeout setting, default value is 8 hours).

In the sendalerts management command we have a main loop that
runs every few seconds and checks for unprocessed flips. The main
loop is unlikely to run into a dead connection issue. The main
loop offloads the actual notification sending to a threadpool
with multiple worker threads. Each thread has its own db connection.
It there's been a long time since the last sent notification then
the thread-specific db connection *can* be already timed out and
closed by the server. Before the fix, in this case the thread crashes
and sendalerts logs a traceback. After the fix, the thread always
runs `close_old_connections()` at the start to make sure it has
a fresh, working db connection.

Fixes: #1267

4 of 8 new or added lines in 3 files covered. (50.0%)

8277 of 9058 relevant lines covered (91.38%)

0.91 hits per line