NVIDIA / nodewright / 37529702502

83%

push

github

fix(operator): leave an untaintable node out of the pass instead of stalling it (#705)

* fix(operator): apply runtime-required taints onto the live taint list

HandleAutoTaint and the reboot re-taint in TrackReboots both patched
spec.taints from the pass's cluster-state snapshot. Taints are an atomic
list, so each write replaced the whole list with the snapshot's copy plus
the runtime-required taint: a taint another controller added after the
snapshot (cluster-autoscaler, Karpenter, the cloud provider, GPU Operator)
was deleted, and one it removed was restored. New and rebooting nodes are
exactly when those controllers are busiest.

Both paths now go through addRuntimeRequiredTaint, which, like
removeRuntimeRequiredTaints, re-reads the node inside each conflict retry
and patches under an optimistic lock, so the taint is appended to the
current list. The reboot path applies the taint before the state reset
and leaves the reboot pending if either write fails, so both are retried
together. It now also records the autoTaint annotation, which is
idempotent for nodes that already carry it.

The heavy pass itself no longer changes taints: runtime-required removal
already re-reads in removeRuntimeRequiredTaints, so these were the
remaining writers.

Fixes #447

Signed-off-by: Alex Yuskauskas <ayuskauskas@nvidia.com>

* fix(operator): re-taint a rebooted node before resetting it, without a marker

The reboot re-taint wrote the autoTaint_<key> annotation, so a node
pre-tainted at provisioning gained a marker the operator never earned,
and a DNS-prefixed taint key (which Validate accepts) produced an invalid
annotation name that the apiserver rejects, leaving the reboot pending
and TrackReboots erroring on every pass. Only HandleAutoTaint marks a
node now; the reboot path relies on the marker surviving Reset, as
before.

The taint write also ran after Reset(), so a failure skipped the node
patch but still saved the wiped status. It now runs first: a failure
lea... (continued)

74 of 80 new or added lines in 3 files covered. (92.5%)

9947 of 11966 relevant lines covered (83.13%)

7.59 hits per line