NVIDIA / nodewright / 41128511140

84%

Build Type

push

github

Committed by web-flow

Commit Message

fix(operator): preserve rebooting nodes in their current batch (#870)

* fix(operator): preserve rebooting nodes in their current batch

Lifecycle taints must not release an active interruption-budget slot while
a node is still down. Keep existing batch members eligible without admitting
already-unreachable nodes that would stall healthy peers.

Preserve explicit Pod eviction timeouts and document the distinction between
fresh admission, batch retention, and eviction.

Closes #868

Signed-off-by: Taylor Daugherty <tdaugherty@nvidia.com>

* fix(operator): exempt lifecycle taints only for a node that holds its batch slot

The exemption keyed off InCurrentBatch, which counts any incomplete
NodePriority entry. A node displaced by an untolerated taint keeps its
entry while another node takes its slot. If it later went unreachable
it was exempt, came back as the only sticky node, and createNewBatch
never ran, so the remaining nodes never started.

HoldsBatchSlot adds the condition that matters: a package stage in
progress on the node. Dispatch writes that entry when it creates the
Job, so a rebooting node always has it and a displaced node never does.

Signed-off-by: Taylor Daugherty <tdaugherty@nvidia.com>

* fix(operator): retain reboot slots across erroring attempts

A reboot can kill the agent before shutdown completes, leaving the package
erroring while its Job still retries. Keep that attempt in its batch so
lifecycle taints cannot admit the next node before recovery.

Pin admission and status reporting in the lifecycle regressions, and make
the documented exemption consistent with explicit tolerations and Pod
eviction timeouts. Terminal-failure slot release remains in #869.

Signed-off-by: Taylor Daugherty <tdaugherty@nvidia.com>

---------

Signed-off-by: Taylor Daugherty <tdaugherty@nvidia.com>
Co-authored-by: Brian Lockwood <lockwobr@gmail.com>
Co-authored-by: ayuskauskas <ayuskauskas@nvidia.com>

Coverage Stats

26 of 28 new or added lines in 2 files covered. (92.86%)

4 existing lines in 1 file now uncovered.

10223 of 12185 relevant lines covered (83.9%)

7.62 hits per line