hypothesis / h
68%
master: 97%

LAST BUILD BRANCH: realtime-update-v1-h-only-1
DEFAULT BRANCH: master
Repo Added
Files 34
Badge
Embed â–¾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

LAST BUILD ON BRANCH ws_exception_handling
branch: ws_exception_handling
CHANGE BRANCH
x

8638

push

travis-ci

nickstenning 
streamer: Don't silently stop processing on exceptions

Prior to this commit, if an exception is thrown by a message handler
(i.e. `h.streamer.broadcast_annotation_message` or
`h.streamer.broadcast_session_change_message`) then the worst possible
combination of events will occur:

1. NSQ will not be notified, and the message will thus not be requeued.
2. The greenlet processing the internal event queue will crash. There is
   no support in the code for restarting this greenlet, and thus all
   WebSocket clients connected to this worker (or that connect
   subsequently) will stop receiving updates.
3. The readers connected to NSQ will continue to receive messages and
   push them onto the internal event queue. These messages will never be
   handled, and will continue to stack up in memory until, presumably,
   the process OOMs.

All in all, not a terribly happy situation.

This commit completely alters how we handle queue messages, fixing all
three issues. The main changes are as follows:

- The internal queue is gone. Instead, we spin up a greenlet on app boot
  which is responsible, in turn, for starting the queue reader
  processes.
- The handling of all messages is done synchronously in the `on_message`
  handlers of the readers. In the event of an exception, `gnsq` will
  handle requeueing the message with NSQ.
- If any reader greenlet terminates (an event which should never occur)
  then all reader greenlets are killed and restarted. A message is also
  written to the logs to indicate that this happened.

There are numerous further simplifications which can now be made to this
code, but I've omitted them from this commit in the interests of making
the changes as clear as possible.

32 of 32 new or added lines in 1 file covered. (100.0%)

2512 of 3677 relevant lines covered (68.32%)

0.68 hits per line

Relevant lines Covered
3677 RELEVANT LINES 2512 COVERED LINES
0.68 HITS PER LINE
Source Files on ws_exception_handling
Detailed source file information is not available for this build.

Recent builds

Builds Branch Commit Type Ran Committer Via Coverage
8638 ws_exception_handling streamer: Don't silently stop processing on exceptions Prior to this commit, if an exception is thrown by a message handler (i.e. `h.streamer.broadcast_annotation_message` or `h.streamer.broadcast_session_change_message`) then the worst possible ... push nickstenning travis-ci pending completion  
8620 ws_exception_handling streamer: Don't silently stop processing on exceptions Prior to this commit, if an exception is thrown by a message handler (i.e. `h.streamer.broadcast_annotation_message` or `h.streamer.broadcast_session_change_message`) then the worst possible ... push nickstenning travis-ci pending completion  
8611 ws_exception_handling streamer: Don't silently stop processing on exceptions Prior to this commit, if an exception is thrown by a message handler (i.e. `h.streamer.broadcast_annotation_message` or `h.streamer.broadcast_session_change_message`) then the worst possible ... push nickstenning travis-ci pending completion  
8576 ws_exception_handling Handle exceptions broadcasting WS messages gracefully If an exception occurs whilst processing an NSQ message in the WebSocket server, log the exception and continue on to the next message. Since there is a single greenlet iterating over all NSQ... push robertknight travis-ci pending completion  
See All Builds (3935)