NVIDIA / nodewright / 42133290152

84%

Build Type

push

github

Committed by web-flow

Commit Message

fix(operator): stop restarting running package steps on unrelated spec edits (#982)

* fix(operator): stop restarting running package steps on unrelated spec edits

Any edit to a NodeWright spec deleted every running package-stage Job and
restarted its step from the beginning. The package init container's
SKYHOOK_DIR is the copy directory, which embeds the CR generation, and
podMatchesPackage left it in the env comparison, so every edit read as spec
drift. SKYHOOK_RESOURCE_ID carries the same generation and was already left
out for exactly this reason.

podMatchesPackage now leaves SKYHOOK_DIR out too. A running step is replaced
only when the edit changes what it runs: the package's version, image,
containerSHA, agentImageOverride, env or resources. That makes the configMap
docs from #942 true: a held change lets the running step finish.

The finished-Job rerun predicate asks the same question, so a stage that has
spent its retries is now released only by those edits as well, and no longer
by any edit as a side effect of the generation. The docs say so.

Closes #978

Signed-off-by: Alex Yuskauskas <ayuskauskas@nvidia.com>

* fix(operator): copy the package in interrupt pods so an edit cannot strand them

Letting a step finish across a spec edit exposed the interrupt pod, which had
no copy step and pointed the agent at the copy directory of the current
generation. After an edit during config nothing had written that directory,
and the agent fails an interrupt whose directory is missing before running it,
so the interrupt crash-looped until it timed out.

Interrupt pods now run the same <package>-init copy as package pods, with the
interrupt container's resources so the pod's requests and limits are
unchanged. Interrupt Jobs are compared by their interrupt container alone, so
a package image edit still leaves a running interrupt in place, and interrupt
Jobs built by an earlier operator still match.

Docs: an interrupt is replaced only by a version or ag... (continued)

Coverage Stats

41 of 41 new or added lines in 1 file covered. (100.0%)

20 existing lines in 2 files now uncovered.

10435 of 12420 relevant lines covered (84.02%)

7.76 hits per line