bradleyfalzon / gopherci

63%

master: 61%

LAST BUILD ON BRANCH pubsub-breaking

branch: pubsub-breaking

CHANGE BRANCH

x

Reset

• pubsub-breaking
• analyser-composition
• build-summary
• builddb
• clone-path
• commit-comment
• debug
• disable-private
• docker
• docs-errors
• dominikh
• dual-comments
• error-handling
• failed-diff
• first-commit
• fix-goveralls
• fix-hardcoded
• gcppubsub
• godep
• gopherci-env
• gopherci.yml
• graceful
• ignore-gen
• integration-test
• logger
• master
• maxcommments
• memlimit
• msgs
• pr-reviews
• pr-reviews-default
• preflight
• pubsub-timeout
• push
• refactor-analyser
• reference-failures
• refreader
• remove-accountid
• reporter
• requeue
• review-no-approve
• senderid
• show-env
• syncronize
• test
• various
• wbsecret
• webui

Build # 171

Build Type

push

travis-ci

Committed by bradleyfalzon

Commit Message

GCPPubSub refactor to support new streaming client API

Google have recently made some breaking changes to their Pub/Sub
client API to support new high performance streaming methods.

https://groups.google.com/forum/#!topic/google-api-go-announce/aaqRDIQ3rvU/discussion
https://groups.google.com/forum/#!topic/google-api-go-announce/8pt6oetAdKc/discussion

The main change is the new methods don't easily support blocking
operations, nor one message at a time use cases. This is being
discussed: https://github.com/GoogleCloudPlatform/google-cloud-go/issues/566

This change attempts to use the new API but in a blocking method
indirectly discussed: https://github.com/GoogleCloudPlatform/google-cloud-go/issues/569

Since finishing and testing this method, which was successful, it
appears Google is discussing this use case further internally, so
this may not be the final solution, but gets us through for the moment.

If we're required to stop using the Pub/Sub client, and instead use
the APIv1 client, the issue does contain a gist of how it could work,
but it hasn't been tested in various failure modes, as it's a lower
level API - but I'm confident it just requires more testing and likely
no more changes.

Further, these changes did necessitate some refectoring on the internal
APIs, this was mostly opportunistic but made the changes simpler.

This refactors were essentially use a channel to push messages onto
the queue, previously this was an interface called Queuer. Also,
previously new jobs to be executed were sent on a channel, instead
each type of queuer should take a callback, and execute that callback
with the job as the only parameter.

A callback was chosen instead of another channel, as I wanted to ensure
only one message was consumed at a time, so the new APIs Receive method
has only one instance running, and calls the callback, blocking until
finished. When using a syncronous channel the Receive method became
asyncronous because it would unblock as ... (continued)

Run Details

581 of 927 relevant lines covered (62.68%)

1.7 hits per line