Unleash / unleash-edge / 39744953810

83%

Build Type

push

github

Committed by web-flow

Commit Message

fix(instance-data): recover from failed upstream posts (#1950)

## About the changes

First step of [fixing the issue with 413s throwing forever](https://linear.app/unleash/issue/EG-4793/make-edge-instance-data-respect-payload-size-limits):

We stop the bleeding on the side of the customer: Edge stops getting stuck when the server rejects a report

Closes #EG-4858


### What was going wrong?

Each Edge Instance sends to Unleash server some metrics every 60''.
A customer’s report got too big (server's limit is at 100 KB on request bodies) and got replied `413 Payload Too Large`.

That's fine. But then Edge did ...nothing. It kept the report it had just failed to send, waited 60'', added everything that had arrived in the meantime, and sent an even bigger one. Then ...a bigger one. Forever, until somebody restarted the process.

### Why it happened?

the code knew what to do about exactly 2 failures:

```rust
StatusCode::NOT_FOUND  => { /* clear the buffer, slow down */ }
StatusCode::FORBIDDEN  => { /* same */ }
_                      => { /* log a warning ... and see ya next time! */ }
```
the `_ ` catches all rest of failures, and `413` fell into it, same would do a `500`, `429`, and every network timeout. 
We didn’t clear the buffer and didn’t increment the counter that controls the retry delay -> so Edge never slowed down. It just kept going with an ever-growing payload.

this was never ONLY a 413 bug.  The customer just happened to find it via 413. Any other failure would have the same stack state.

### And now?

We give every failure a certain behavior (same way as the other metrics in Edge work):

- 413 too large	= > throw the buffer away. No slow down. - It’ll never fit. Dropping it is what makes the next one smaller
- 400 bad request	=> throw it away	- It’ll never be accepted either
- 403 / 404	=> no change by purpose
- 429, 5xx, timeouts	=> keep buffer, slow down	- their problem, probably temporary, the data is still good
- 200 => buffer empty... (continued)

Coverage Stats

106 of 113 new or added lines in 1 file covered. (93.81%)

7 existing lines in 2 files now uncovered.

14089 of 16998 relevant lines covered (82.89%)

4472.29 hits per line