Logflare / logflare / 3a56c0e53fc0cdbd02b51417a0ca9b717084835b-PR-3626

82%

main: 82%

Build Type

Pull #3626

github

Committed by djwhitt

Commit Message

refactor(ingest queue): timestamp-based stale :processing recovery

Replace the snapshot/intersection stale-detection in QueueJanitor with a
per-row monotonic claim timestamp. Each queue row gains a sixth field,
claimed_at, set to System.monotonic_time(:millisecond) when a row is claimed
into :processing and reset to 0 on every transition back out. The janitor now
selects :processing rows whose claimed_at is older than a fixed staleness age
instead of building and diffing MapSets of in-flight IDs each cycle.

Stale cleanup is treated as crash recovery rather than normal queue
maintenance: the cadence is slowed to 60s, rows are considered stale after
120s, and each pass is bounded to 1000 rows per queue (logging when the cap is
hit). The retry-then-drop ladder and the select_replace CAS guard are
unchanged.

The claim path reads the clock once per take_pending_ids batch (not per
event) and stamps every row in the batch with the same value. No separate
processing_count is maintained; the bounded select runs each pass.

Status is pinned to :processing in the stale match head, so pending/ingested
rows (claimed_at == 0) are never selected regardless of the monotonic clock's
sign — avoiding a sentinel guard that would misbehave with negative monotonic
time.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Pull Request Pull Request #3626: refactor(ingest queue): timestamp-based stale :processing recovery

Coverage Stats

29 of 34 new or added lines in 3 files covered. (85.29%)

11 existing lines in 5 files now uncovered.

13327 of 16368 relevant lines covered (81.42%)

5139.36 hits per line