lsm / HyperNeo / 31984369412

79%

Build Type

push

github

Committed by web-flow

Commit Message

fix(daemon): jitter stale-reclaim re-enqueue to prevent synchronized SDK cold-start herd (#2550)

* fix(daemon): jitter stale-reclaim re-enqueue to prevent synchronized SDK cold-start herd

When the daemon dies with M message_delivery jobs in flight, their rows
freeze in `processing` until the leases go stale (5 min), and stale-reclaim
then re-enqueues ALL of them in the same instant — measured 10 claims within
12 ms in the live repro. Each reclaimed job spawns a fresh SDK subprocess
resuming a large transcript; M simultaneous cold-starts all exceed the 15 s
startup timeout (a solo cold-start is ~4–5 s), producing a self-sustaining
timeout/retry loop (120–160 timeouts, 0 successes over 7+ min).

The reclaim pass now spreads each re-enqueued job's run_at across a
randomized [0, min(M*2s, 30s)] window, realized as M equal slots with a
randomly permuted slot per job plus a random in-slot offset: the order is
random per pass (restarts never produce a deterministic stagger) while the
density is bounded by construction — no 1-second window schedules more than
3 reclaimed jobs through M ~ 60. A single reclaimed job gets no delay, so a
lone stuck job still recovers immediately.

* fix(daemon): jitter graceful-shutdown restart herd; harden reclaim jitter path

Review follow-ups on PR #2550:

- P2: the SIGTERM path (app.ts requeueAllProcessing at run_at=now) produced
  the deploy-path twin of the crash herd — the next boot's first tick claimed
  the whole pending fleet at once, since eager reclaim only sweeps
  `processing` rows. requeueAllProcessing now returns the re-enqueued job
  IDs, and the shutdown path spreads them with the same
  staleReclaimJitterDelays + reschedulePending primitives; a single requeued
  job keeps zero delay (#2593's prompt recovery unchanged). Docs bullets
  cross-reference instead of contradicting.
- P3: reschedule loop wrapped in try/catch so a failed UPDATE can never skip
  the abort/deferral loop (un-jittered rows fall back to im... (continued)

Coverage Stats

43577 of 59186 branches covered (73.63%)

Branch coverage included in aggregate %.

43 of 46 new or added lines in 3 files covered. (93.48%)

2 existing lines in 1 file now uncovered.

59350 of 71296 relevant lines covered (83.24%)

3800.28 hits per line