GoogleCloudPlatform / google-cloud-java / 4275

72%

Pull #2281

travis-ci

google-cloud-nio: retry in more situations upon retryable errors

There were a number of operations in google-cloud-nio that were prone
to intermittent failures (such as 503 errors) when run at scale, but
which were not wrapped in a retry loop or hooked up to the client settings
for number of retries/reopens. This rendered the google-cloud-nio library
effectively unusable at scale.

This commit wraps those additional operations in CloudStorageReadChannel
and CloudStorageFileSystemProvider within retry/reopen loops, with the number
of retries/reopens determined by the "maxChannelReopens" client settings.

If the client has not set "maxChannelReopens", they'll still get a nice
error message with the underlying error (such as a 503), plus an indication
that 0 retries/reopens were attempted.

To facilitate this change, we've added a CloudStorageRetryHandler that
keeps track of the total number of retry/reopen attempts, and throws
an appropriate exception when out of attempts.

Note that this change doesn't affect the normal retries done at a
lower level by the cloud.storage.Storage object.

122 of 122 new or added lines in 4 files covered. (100.0%)

19882 of 24995 relevant lines covered (79.54%)

56986.29 hits per line