NVIDIA / nodewright / 37527108213

83%

Build Type

push

github

Committed by web-flow

Commit Message

fix(operator): do not take ownership of a cordon NodeWright did not create (#760)

Cordon() wrote the cordon_<name> ownership annotation whenever it was
missing, even on a node that was already unschedulable. A cordon placed by
another controller was therefore recorded as NodeWright's, and when the
package completed Uncordon() cleared spec.unschedulable. With NVSentinel on
GKE COS this released nodes mid health check: NVSentinel cordons and
reboots, REAPPLY_ON_REBOOT reruns the tuning package, its service interrupt
claims the cordon, and completion uncordons a node that may still be
unhealthy.

Cordon() now leaves a node alone when it is unschedulable and no NodeWright
holds a cordon on it (no cordon_* key, no runtimeRequiredCordon), and
returns false since that cordon is already durable in the API, so the drain
starts in the same pass. A node where another NodeWright holds a cordon is
handled as before, and if the external cordon is lifted mid-run the next
Cordon() cordons the node itself and owns it.

The reboot Reset() no longer deletes cordon_<name>. Its only caller is
TrackReboots, and a NodeWright's own reboot interrupt changes the boot ID
while the node is still cordoned; dropping the annotation there would leave
a cordon no NodeWright holds, which Cordon() now refuses to take, stranding
the node. The TrackReboots envtest spec asserted the annotation was
cleared; it now asserts it survives.

The CLI reset commands had the same problem: they strip cordon_<name>
without uncordoning, and the documented drain-timeout recovery relies on
the re-run releasing the cordon. `reset` and `node reset` now keep the
annotation on nodes the NodeWright still selects, matching its
nodeSelector the way the operator does, and remove it where no re-run
will come: the NodeWright is gone (the orphaned-cordon case) or no
longer selects the node. If the NodeWright cannot be read the
annotation is kept with a warning. A node whose only remaining metadata is a kept
cordo... (continued)

Coverage Stats

50 of 53 new or added lines in 4 files covered. (94.34%)

13 existing lines in 2 files now uncovered.

9893 of 11917 relevant lines covered (83.02%)

7.57 hits per line