prisma-risk / tsoracle / 26249338202

95%

push

github

fix(stress): make supervisor monotonicity check arrival-order-tolerant (#43)

* fix(stress): make supervisor monotonicity check arrival-order-tolerant

The supervisor's "global monotonicity" check compared each `IssuedSample`
against a global `high_water` updated in mpsc arrival order. With multiple
concurrent client tasks all producing into the same supervisor channel,
arrival order at the supervisor is not issue order at the TSO. Under the
raft + killer-loop scenario, 8 stalled RPCs can unblock together with
contiguous ts assigned monotonically by the new leader yet land at the
supervisor in an arbitrary permutation — a legitimate observation, not a
TSO regression. The race is widened by `cargo llvm-cov` instrumentation,
which is why it manifested on the coverage CI job.

Replace the global arrival-order check with two client-observable
invariants: per-client monotonicity (a single client's sequential
`await`-ed RPCs must strictly increase) and cross-client uniqueness (the
TSO must never issue the same ts twice). The global `high_water` field
is kept as max-observed and continues to feed the FenceFreshness check,
which is the right place for "no regression across a leadership change".

Add a regression test that reproduces the failure shape (Client 7's
ts=551 arriving before Clients 0..6's ts=544..550) and asserts no
violations.

* fix(stress): attribute DeadlineExceeded to chaos via retry-interval overlap

PR #43's first commit fixed the Monotonicity flake, but the same smoke
test then hit a Liveness flake on CI: under `killer-loop` (KillLeader
every 2s), 8 clients each retried for the 5s `liveness_deadline` while
the cluster churned through back-to-back leader elections, then all
deadlined with "FailedPrecondition: not leader" — yet the supervisor
reported it as a TSO liveness regression rather than chaos-induced
unavailability.

`on_liveness` previously checked whether `incident.at` (the instant the
deadline fired) fell inside an open chaos windo... (continued)

136 of 136 new or added lines in 1 file covered. (100.0%)

5442 of 5705 relevant lines covered (95.39%)

96775.11 hits per line