hypothesis / h / 8638

68%

master: 97%

push

travis-ci

streamer: Don't silently stop processing on exceptions

Prior to this commit, if an exception is thrown by a message handler
(i.e. `h.streamer.broadcast_annotation_message` or
`h.streamer.broadcast_session_change_message`) then the worst possible
combination of events will occur:

1. NSQ will not be notified, and the message will thus not be requeued.
2. The greenlet processing the internal event queue will crash. There is
   no support in the code for restarting this greenlet, and thus all
   WebSocket clients connected to this worker (or that connect
   subsequently) will stop receiving updates.
3. The readers connected to NSQ will continue to receive messages and
   push them onto the internal event queue. These messages will never be
   handled, and will continue to stack up in memory until, presumably,
   the process OOMs.

All in all, not a terribly happy situation.

This commit completely alters how we handle queue messages, fixing all
three issues. The main changes are as follows:

- The internal queue is gone. Instead, we spin up a greenlet on app boot
  which is responsible, in turn, for starting the queue reader
  processes.
- The handling of all messages is done synchronously in the `on_message`
  handlers of the readers. In the event of an exception, `gnsq` will
  handle requeueing the message with NSQ.
- If any reader greenlet terminates (an event which should never occur)
  then all reader greenlets are killed and restarted. A message is also
  written to the logs to indicate that this happened.

There are numerous further simplifications which can now be made to this
code, but I've omitted them from this commit in the interests of making
the changes as clear as possible.

32 of 32 new or added lines in 1 file covered. (100.0%)

2512 of 3677 relevant lines covered (68.32%)

0.68 hits per line