bedrock-kv / bedrock / 8415dbf89a1b79e0bb2b2ae7fae2c0b37488faf6

82%

Build Type

push

github

Committed by web-flow

Commit Message

Commit path: full-jitter backoff, ETS routing cache, adaptive batching (#209)

Ticket: `bedrock-62e`. Found while chasing a reported slowdown in the
class-scheduling demo's "dans" section.

## The defect

```elixir
base_delay = 1 <<< retry_count
jitter = :rand.uniform(3)          # one to three milliseconds
```

A ceiling that doubles, with 1–3ms of jitter. Transactions contending on
one key fail at nearly the same instant, compute nearly the same delay,
and **wake together** — so they collide again, and the round trip
repeats at twice the delay. The work is trivial; the wall clock is set
by whichever contender climbs furthest up the ladder.

## Measured

100 concurrent signups against one seat counter, ten independent rounds,
fresh class each round:

| | median | **max** | total |
|---|---|---|---|
| before | 285ms / 540ms | **1058ms / 3057ms** | 4761ms / 7161ms |
| after | 210ms / 202ms | **387ms / 320ms** | 2327ms / 2054ms |

Before, samples snap to a ladder — `280, 540, 1058, 3057` — every
contender climbing the same rungs. After, they spread and the tail
collapses **2.7–9.5×**.

## The fix is FDB's

Full jitter: the delay is a uniform draw from the **whole** interval
below the ceiling, and only the *ceiling* grows.

- `NativeAPI.actor.cpp:4436` — `returnedBackoff *=
deterministicRandom()->random01();`
- `:4446` — `backoff = std::min(backoff *
CLIENT_KNOBS->BACKOFF_GROWTH_RATE, ...)`
- `ClientKnobs.cpp:67-69` — 10ms initial, 1s max, growth 2.0

The ceiling schedule here is **unchanged**. What changed is drawing from
all of it rather than a 3ms band at its top. Never zero, so a retry
cannot become a spin.

## Not a regression

Bisected: 198ms at `0.5.3`, 150ms at `0.6.0`, 283ms at `0.6.1`-era,
277ms pre-merge, 281ms today. Pickaxe dates the schedule to `602f0c8a`.
It has always been this way and was simply never measured.

## Tests

`retry_delay_in_ms/1` extracted as a pure function and tested for
properties rather than values: ceiling still double... (continued)

Coverage Stats

30 of 36 new or added lines in 6 files covered. (83.33%)

2 existing lines in 1 file now uncovered.

6770 of 8653 relevant lines covered (78.24%)

1033.64 hits per line