NVIDIA / nodewright / 36903455612

83%

Build Type

push

github

Committed by web-flow

Commit Message

fix(operator): keep sticky batch nodes running while others are in progress (#698)

* fix(operator): keep sticky batch nodes running while others are in progress

GetNodesForNextBatch returned only the InProgress nodes whenever any node
in the compartment was InProgress, so a batch member sitting at Waiting
between packages (e.g. interrupt complete, post-interrupt pending) was
never selected until the whole compartment went idle. With a large budget
and long-running stages that may never happen, and post-interrupt starves.

Return the InProgress nodes together with the sticky NodePriority members.
getStickyBatchNodes now skips InProgress nodes so the two sets are
disjoint. The union keeps InProgress nodes that lack a NodePriority entry,
which reordering the sticky check first would drop. Batch membership is
unchanged: every sticky node was already admitted to the batch.

Fixes #587

Signed-off-by: Alex Yuskauskas <ayuskauskas@nvidia.com>

* fix(operator): cap sticky nodes at the batch size while others run

A node that was ignored, untolerated or relabelled in keeps its
NodePriority entry while another node takes its slot, so the entries can
outnumber the budget. Returning every sticky node alongside the
InProgress ones then ran more nodes than the budget allows. While nodes
are InProgress, admit sticky nodes only up to the batch size. With
nothing InProgress they are still returned uncapped, as before, so an
erroring sticky node cannot hold the only slot.

Collect InProgress and sticky nodes in one loop instead of two helpers
coupled by a duplicate guard.

Signed-off-by: Alex Yuskauskas <ayuskauskas@nvidia.com>

* fix(operator): apply one package per node per pass under serial

With spec.serial, RunSkyhookPackages returned after the first
ApplyPackage of the pass. RunNext includes packages already running, so
the first selected node was reapplied every pass and no other node
advanced until it finished: serial ran one node at a time, contrary to
custo... (continued)

Coverage Stats

22 of 22 new or added lines in 2 files covered. (100.0%)

5 existing lines in 1 file now uncovered.

9509 of 11601 relevant lines covered (81.97%)

7.62 hits per line