soapiestwaffles / s3-nuke / 9260420326

52%

push

github

Implement rate limiting retryer

When receiving a rate limit response from AWS, the application would
exit with an error. This was an undesirable outcome.

AWS has a feature to implement retrying using their `retry`
[package][1].

By default there are a number of retryable conditions. The relevant
useful cases are as follows:

- Connection Errors
- RequestTimeout, RequestTimeoutException
- Throttling, ThrottlingException, ThrottledException,
  RequestThrottledException, TooManyRequestsException,
  RequestThrottled, SlowDown
- RequestLimitExceeded, BandwidthLimitExceeded, LimitExceededException

This means the default retryer will handle rate limiting and will not
need to implicitly handle this case.

It is also important to note the following about client rate limiting:

> Generally you will always want to return new instance of a Retryer.
> This will avoid a global rate limit bucket being shared across all
> service clients.

This means that the instantiation of the S3 client must be handled
within each Go routine and cannot be shared as it was previously
implemented.

`AdaptiveMode` is the retry strategy that will be used:

> AdaptiveMode provides an experimental retry strategy that expands on
> the Standard retry strategy, adding client attempt rate limits. The
> attempt rate limit is initially unrestricted, but becomes restricted
> when the attempt fails with for a throttle error.

The default, values for the `AdaptiveMode` is based on the
`NewStandard` which is:

- `DefaultMaxAttempts`: `3`
- `DefaultMaxBackoff`: `20s`

[1]: https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/aws/retry

Fixes: #21
Signed-off-by: Michael Lorant <michael.lorant@nine.com.au>

4 of 14 new or added lines in 2 files covered. (28.57%)

391 of 898 relevant lines covered (43.54%)

0.5 hits per line