Freegle / Iznik / 21079

71%

push

circleci

fix(reaper): never reap a VM whose runner is actively connected (reused-VM mid-job bug)

Root cause of the residual ~31% build-test-katapult infrastructure_fails
(2026-06-13): the reaper keys "is this VM orphaned?" on the pipeline number baked
into the VM NAME — i.e. the pipeline that PROVISIONED it. But runner VMs are
pooled and reused, so a VM named for pipeline A is frequently running pipeline B's
job. Once A finishes, the reaper considers the VM orphaned and DELETEs it via the
Katapult API — killing B's job mid-run -> infrastructure_fail. The only guard was
MIN_AGE_SECONDS, and a stale deployment still running the old MIN_AGE=1800 (raised
to 9000 in d60571507) reaps reused VMs ~30 min in. Evidence: Katapult trash shows
deletions at HH:MM:02 (per-minute cron) at ~30-34 min age once the name-pipeline
finished, while a VM whose own pipeline was still running survived to 48 min.

Fix: independently of age/pipeline, refuse to delete a VM whose CircleCI runner
(named after the VM) connected within RUNNER_ACTIVE_GRACE_SECONDS (300s). A runner
actively running a task holds a continuous connection, so last_connected is always
a few seconds old — verified live: a busy VM reads last_connected ~1s, idle/done
VMs read minutes. This protects busy VMs even if MIN_AGE is misconfigured low, so
the bug cannot recur regardless of which deployment runs. Fail safe: if the runner
inventory can't be fetched, treat all as active and delete nothing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

10705 of 14260 branches covered (75.07%)

Branch coverage included in aggregate %.

116840 of 165323 relevant lines covered (70.67%)

35.85 hits per line