stacklok / toolhive / 25866747879

66%

Build Type

push

github

Committed by web-flow

Commit Message

Reset RunWorkload retry counter after stable run (#5172)

* Reset RunWorkload retry counter after stable run

When the Docker daemon is repeatedly unresponsive, the RunWorkload
retry loop's 10-attempt budget can be exhausted by short-lived but
successful starts: the container exits, the manager retries, the new
container runs healthily for tens of seconds before the next
disruption kills it, and the cycle repeats. After 10 cycles the
manager logs "failed to restart after max attempts, giving up" and
the workload stays offline until manually restarted, even though
the original cause was transient.

Track the duration of each runner.Run call. If the workload ran past
a stable threshold before failing, treat the next attempt as a fresh
failure rather than a continuation of the current retry sequence:
reset the attempt counter and backoff delay. This preserves the
budget's role as a bound on tight crash loops while allowing
recovery from sustained external disturbances.

Add a small testability seam (runner factory on DefaultManager,
defaulting to runner.NewRunner) so the retry loop has unit-test
coverage of the boundary cases.

Fixes #5171

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Greg Katz <gkatz@indeed.com>

* Use functional options for retry test injection

Replace direct struct-field assignment in tests with an unexported
managerOption type and two helper functions (withRetryConfig and
withRunnerFactory). The struct fields stop advertising test-only intent
in their comments; the options carry that intent now.

Also rewrite the retry loop with an explicit attempt counter so the
reset and increment paths do not rely on the for-clause's implicit
increment.

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Greg Katz <gkatz@indeed.com>

---------

Signed-off-by: Greg Katz <gkatz@indeed.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Muhammad Amir Ejaz <amir@stacklok.com>

Coverage Stats

50 of 51 new or added lines in 1 file covered. (98.04%)

35 existing lines in 5 files now uncovered.

64695 of 98971 relevant lines covered (65.37%)

62.56 hits per line