cloudmarker / cloudmarker / 744

91%

master: 84%

Pull #128

travis-ci-com

Do not let workers crash on plugin errors

This change improves the error handling in the `workers` module. Every
plugin method invocation is wrapped with a `try`-`except` block to
handle errors from the plugin method gracefully.

It is especially important to prevent store, event, and alert workers
from crashing because they consume data from an input queue. If one or
more of these workers crash, then they would no longer read data from
their queues. As other plugins such as cloud or event plugins write more
data to these queues, the underlying OS pipe would become full. After
that the cloud or event plugins would block and the framework would
hang.

This change is about preparing for the worst. Ideally, the workers
should be written carefully such that they never lead to an unhandled
error. However, since writing a 100% error-free plugin can never be
guaranteed, and more importantly, since the plugin's robustness is
beyond the framework's control, when it comes to the worst and a plugin
raises an error, the framework handles it and moves on. Especially, this
change ensures that the consumer workers can continue to consume data
from their input queues despite plugin errors.

Since unhandled plugin errors would be really unexpected and
exceptional, this change logs the complete exception details along with
the stack trace, so that it can be used to investigate the issue later.

341 of 388 branches covered (87.89%)

Branch coverage included in aggregate %.

30 of 30 new or added lines in 1 file covered. (100.0%)

1667 of 1842 relevant lines covered (90.5%)

2.71 hits per line