Freegle / Iznik / 22239

71%

push

circleci

fix(ci): reap zombie-pinned runner VMs (stuck-'running' workflow fallback) (#807)

A stuck/orphaned CircleCI workflow stays in 'running' state indefinitely
(observed 18h+ on PR #789: pipeline 7925's build-test-katapult never
reported a terminal status). The reaper checked pipeline_is_running()
FIRST and bailed (`continue`), so the VM named after that pipeline was
pinned — and billed on an expensive ROCK-48 package — forever.

Add a zombie fallback that runs before the age/pipeline checks and keys on
the runner's connection, which is the ground truth: a runner executing a
task (or idle in the pool) heartbeats continuously, so a last_connected
older than ZOMBIE_RUNNER_IDLE_SECONDS (1h) means the runner process is dead
and nothing is running — regardless of what the workflow status claims.
Such a VM is reaped even if its pipeline is wedged in 'running', and even
below MIN_AGE (the stale-heartbeat requirement makes that safe).

Safety:
- runner_is_zombie_idle() fires only on POSITIVE evidence (a real but stale
  last_connected); a booting/not-yet-dispatched runner (no entry) or a
  missing inventory both return false, so a VM waiting to start its job is
  never reaped.
- 1h threshold is 3x the busiest idle heartbeat gap observed on live healthy
  VMs (0-21min), so a quiet test phase or CPU-starvation blip can't trip it.
- Runner inventory now keeps the FRESHEST last_connected per name: a single
  live registration protects a VM even if CircleCI briefly lists a stale
  duplicate alongside it (this path no longer has pipeline_is_running as a
  backstop, so the de-dupe matters).

Validated by repeated --dry-run against live Katapult+CircleCI: zero healthy
VMs flagged at 3600s; the zombie branch fires and logs correctly when the
threshold is lowered.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

11012 of 14630 branches covered (75.27%)

Branch coverage included in aggregate %.

119016 of 168945 relevant lines covered (70.45%)

36.26 hits per line