bradleyfalzon / gopherci / 168

63%

master: 61%

push

travis-ci

GCPPubSub refactor to support new streaming client API

Google have recently made some breaking changes to their Pub/Sub
client API to support new high performance streaming methods.

https://groups.google.com/forum/#!topic/google-api-go-announce/aaqRDIQ3rvU/discussion
https://groups.google.com/forum/#!topic/google-api-go-announce/8pt6oetAdKc/discussion

The main change is the new methods don't easily support blocking
operations, nor one message at a time use cases. This is being
discussed: https://github.com/GoogleCloudPlatform/google-cloud-go/issues/566

This change attempts to use the new API but in a blocking method
indirectly discussed: https://github.com/GoogleCloudPlatform/google-cloud-go/issues/569

Since finishing and testing this method, which was successful, it
appears Google is discussing this use case further internally, so
this may not be the final solution, but gets us through for the moment.

If we're required to stop using the Pub/Sub client, and instead use
the APIv1 client, the issue does contain a gist of how it could work,
but it hasn't been tested in various failure modes, as it's a lower
level API - but I'm confident it just requires more testing and likely
no more changes.

Further, these changes did necessitate some refectoring on the internal
APIs, this was mostly opportunistic but made the changes simpler.

This refactors were essentially use a channel to push messages onto
the queue, previously this was an interface called Queuer. Also,
previously new jobs to be executed were sent on a channel, instead
each type of queuer should take a callback, and execute that callback
with the job as the only parameter.

GCPPubSubQueue was tested to ensure only one message is removed from
the queue at any time, allowing other instances to consume the remaining
messages, and on shutdown the executing job is allowed to finish in
full before the process exits.

117 of 117 new or added lines in 5 files covered. (100.0%)

590 of 927 relevant lines covered (63.65%)

1.71 hits per line