NVIDIA / nodewright / 37505183488

83%

Build Type

push

github

Committed by web-flow

Commit Message

fix(operator): end a crash-looping non-reboot interrupt after jobBackoffLimit restarts (#745)

Interrupt Jobs restart the interrupt in place with no retry limit,
because a reboot interrupt's own shutdown kills its container. Since
interrupt held its cordoned, drained node in_progress for the full
jobStageTimeout (1h by default), or forever with no deadline, and its
DeploymentPolicy batch was never evaluated.

Non-reboot interrupt Jobs now carry their restart allowance
(JOB_BACKOFF_LIMIT), stamped at creation. When the interrupt container
is genuinely failing with a restart count at or above it, the pod watch
re-reads the pod uncached and sets the Job's backoffLimit to 0 with a
restart-limit-exceeded annotation, emitting a RestartLimitExceeded
Warning event on the Node and the NodeWright. The Job controller then
fails the Job and the existing failed-Job path marks the node erroring.
jobFailureIsGenuine accepts the annotation, since the pod is gone by
then. Reboot interrupts are not stamped and keep relying on the stage
deadline, so a slow but healthy shutdown is never failed. Suspended,
invalid, already-failing and reset Jobs are skipped.

A failed Job's last-logs snapshot now captures the output of a
container in CrashLoopBackOff instead of the waiting reason. Docs, the
chart, the StageTimeout field doc and the release notes describe the
new bound and its limits.

Fixes #738

Signed-off-by: Brian Lockwood <lockwobr@gmail.com>

Coverage Stats

139 of 175 new or added lines in 4 files covered. (79.43%)

6 existing lines in 1 file now uncovered.

9849 of 11873 relevant lines covered (82.95%)

7.59 hits per line