uber / cadence / 019060e3-e556-4061-8a61-c080801be36f / 1
72%
master: 72%

DEFAULT BRANCH: master
Ran
Files 727
Run time 16s
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 71.523% (+0.09%) from 71.438%
019060e3-e556-4061-8a61-c080801be36f.1

push

buildkite

web-flow 
Global ratelimiter: everything else (#6141)

After too many attempts to break this apart and build different portions in self-contained ways, and running into various inter-dependent roadblocks... I just gave up and did it all at once.

# Rollout plan for people who don't want or need this system

Do nothing :)

As of this PR, you'll use "disabled" and that should be as close to "no changes at all" as possible.
Soon, you'll get "local", and then you'll have some new metrics you can use (or ignore) but otherwise no behavior changes.

And that'll be it.  The "global" load-balanced stuff is likely to remain opt-in.

# Rollout plan for us

For deployment: any order is fine / should not behave (too) badly.  Even if "global" or either shadow mode is selected on the initial deploy.  Frontends will have background `RatelimitUpdate` request failures until History is deployed, but that'll just mean it continues to use the "local" internal fallback and that's in practice the same behavior as "local" or "disabled", just slightly noisier.

The _smoothest_ deployment is: deploy everything on "disabled" or "local" (the default(s), so no requests are sent until deploy is done), then switch to "local-shadow-global" to warm global limiters / check that it's working, then "global" to use the global behavior.  

Rolling back is just the opposite.  Ideally disable things first to stop the requests, but even if you don't it should be fine.

In more detail:

1. At merge time, this will set the "key mode" (`frontend.globalRatelimiterMode`) to "disabled", which gets as close as is reasonably possible to acting _exactly_ like it did before this PR.
   - This is also effectively the panic button for the initial rollout.
2. Once that proves to not immediately explode, switch to "local" for all keys.  This will keep the current ratelimiter rates, but will start collecting and emitting ratelimiter-usage metrics, so we can make sure that doesn't explode eithe... (continued)

105319 of 147252 relevant lines covered (71.52%)

2630.59 hits per line

Source Files on job 019060e3-e556-4061-8a61-c080801be36f.1