NVIDIA / nodewright / 36497038583

83%

Build Type

push

github

Committed by web-flow

Commit Message

fix(operator): keep taint-blocked nodes blocked while sequencing holds them (#700)

* fix(operator): keep taint-blocked nodes blocked while sequencing holds them

A node with a taint the NodeWright does not tolerate, and held at waiting
by cross-NodeWright sequencing, flipped every reconcile: IntrospectNode
set it to waiting because IsNodeReadyForSkyhook rejected it, then the
selection scan set it back to blocked. The net status never changed, but
each flip latched the node as changed, so every pass issued a no-op Node
PATCH and requeued at 2s, per affected node, indefinitely.

IntrospectNode now treats untolerated taints like the ignore label: a
non-complete node is blocked before the sequencing check, using the same
tolerations as SelectNodes (extracted into packageTolerations). Both
Introspect functions take the runtime-required tolerations as a parameter
to build them.

Blocked is now re-derived like waiting, paused and disabled once its
cause is gone. Without that, a node blocked while its NodeWright is held
by a sequencing: all predecessor stays blocked after the taint or ignore
label is removed, because nothing selects it until the predecessor
finishes.

Fixes #603

Signed-off-by: Alex Yuskauskas <ayuskauskas@nvidia.com>

* fix(operator): judge node taints by the tolerations package pods carry

Node selection built its own toleration list: the cordon taint for
NoSchedule only, and the runtime-required taints only when the
NodeWright set runtimeRequired. The job builder stamps every package pod
with the cordon taint for any effect and the runtime-required taints
unconditionally, and taints.md documents both as always tolerated. Now
that IntrospectNode reuses the list on every reconcile, the drift showed:
a plain NodeWright behind a runtime-required one reported a freshly
auto-tainted node as blocked, with no TaintNotTolerable condition, though
its pods would schedule there.

packageTolerations in job_builder.go is now the single list, used by
bo... (continued)

Coverage Stats

18 of 18 new or added lines in 3 files covered. (100.0%)

13 existing lines in 2 files now uncovered.

9353 of 11345 relevant lines covered (82.44%)

7.64 hits per line