cloudmarker / cloudmarker / 756

84%

push

travis-ci-com

Do not let workers crash on plugin errors

This change improves the error handling in the `workers` module. Every
plugin method invocation is wrapped with a `try`-`except` block to
handle errors from the plugin method gracefully.

It is especially important to prevent store, event, and alert workers
from crashing because they consume data from an input queue. If one or
more of these workers crash, then they would no longer read data from
their queues. As other plugins such as cloud or event plugins write more
data to these queues, the underlying OS pipe would become full. After
that the cloud or event plugins would block and the framework would
hang.

This change is about preparing for the worst. Ideally, the workers
should be written carefully such that they never lead to an unhandled
error. However, since writing a 100% error-free plugin can never be
guaranteed, and more importantly, since the plugin's robustness is
beyond the framework's control, when it comes to the worst and a plugin
raises an error, the framework handles it and moves on. Especially, this
change ensures that the consumer workers can continue to consume data
from their input queues despite plugin errors.

Since unhandled plugin errors would be really unexpected and
exceptional, this change logs the complete exception details along with
the stack trace, so that it can be used to investigate the issue later.

372 of 419 branches covered (88.78%)

Branch coverage included in aggregate %.

1874 of 2049 relevant lines covered (91.46%)

2.74 hits per line