NVIDIA / nodewright / 38597705289

84%

Build Type

push

github

Committed by web-flow

Commit Message

fix(operator): only reset complete or erroring nodes on config update (#788)

* fix(operator): only reset complete or erroring nodes on config update

HandleConfigUpdates moved every node matching the selector back to the
config stage when a package ConfigMap changed. That interrupted nodes
mid-apply or mid-upgrade and invalidated their running jobs as stale.

Reset only nodes that are complete with no running job, or erroring in
config, interrupt or post-interrupt. Other nodes reach config on their
own with the updated ConfigMap.

Part of #776.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Noah Kontur <nkontur@nvidia.com>

* fix(operator): reset every node at or past config on config update

Address review on #788. Skipping nodes already past config let them
finish on the old ConfigMap, since the ConfigMap is updated once and no
later reconcile sees a diff.

Once the gate opens (all nodes complete, or one erroring at config,
interrupt or post-interrupt), reset every node at config, interrupt or
post-interrupt, plus complete ones, and clear their unfinished or timed-out
executors. Nodes without the package or at uninstall, apply or upgrade are
left alone and reach config on their own.

Tests cover each stage/state the review listed plus the closed gate.

Part of #776.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Noah Kontur <nkontur@nvidia.com>

* fix(operator): clear an unrecorded config Job when resetting a node

Address CodeRabbit on #788. A config Job that finished successfully but whose
completion was not yet recorded survived the reset. With the entry back at
config, JobReconciler would record it as the new run's completion, and the
node would finish on the old ConfigMap with no diff left to retry.

Foreground-delete such a Job together with the unfinished and timed-out ones;
a terminating Job is never recorded. Other stages' retained Jobs stay.

Part of #776.

Co-Authored-By: Claude Sonne... (continued)

Coverage Stats

18 of 21 new or added lines in 1 file covered. (85.71%)

3 existing lines in 1 file now uncovered.

9970 of 11981 relevant lines covered (83.22%)

7.6 hits per line