Logflare / logflare / 98fd8073677eb25668d77c26241056ce5c355eae

82%

Build Type

push

github

Committed by web-flow

Commit Message

refactor(ingest queue): timestamp-based stale :processing recovery (#3626)

* refactor(ingest queue): timestamp-based stale :processing recovery

Replace the snapshot/intersection stale-detection in QueueJanitor with a
per-row monotonic claim timestamp. Each queue row gains a sixth field,
claimed_at, set to System.monotonic_time(:millisecond) when a row is claimed
into :processing and reset to 0 on every transition back out. The janitor now
selects :processing rows whose claimed_at is older than a fixed staleness age
instead of building and diffing MapSets of in-flight IDs each cycle.

Stale cleanup is treated as crash recovery rather than normal queue
maintenance: the cadence is slowed to 60s, rows are considered stale after
120s, and each pass is bounded to 1000 rows per queue (logging when the cap is
hit). The retry-then-drop ladder and the select_replace CAS guard are
unchanged.

The claim path reads the clock once per take_pending_ids batch (not per
event) and stamps every row in the batch with the same value. No separate
processing_count is maintained; the bounded select runs each pass.

Status is pinned to :processing in the stale match head, so pending/ingested
rows (claimed_at == 0) are never selected regardless of the monotonic clock's
sign — avoiding a sentinel guard that would misbehave with negative monotonic
time.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(queue janitor): CAS-protect the stale drop path

The drop branch of act_on_stale_event matched any :processing row with the
target id and ignored select_delete's return, so it reported :drop even when
nothing was deleted (inflating the dropped data-loss metric) and could, in the
narrow lookup->delete window, delete a row that had been acked and re-claimed
under the same id.

Pin the exact row observed by the lookup as the CAS match in both the drop and
reset branches (which also guards claimed_at, so a freshly re-claimed row no
longer matches) and gate... (continued)

Coverage Stats

39 of 39 new or added lines in 3 files covered. (100.0%)

1 existing line in 1 file now uncovered.

13414 of 16439 relevant lines covered (81.6%)

5167.36 hits per line