NVIDIA / nodewright / 36897104756

push

github

fix(operator): mark a node erroring only when a stage fails (#692)

* fix(operator): clear a node's stale erroring status once its interrupt completed

A native reboot interrupt whose container is terminated during shutdown (exit
143) left the node reporting `erroring` after it came back Ready, with its
package already at `interrupt/complete`. Post-interrupt work never ran, the node
stayed cordoned, and the rollout stayed stopped.

Two things kept it stuck, and both are fixed here:

`IntrospectNode` now clears the stale node-level `erroring` status back to
`waiting` when the node is kubelet-Ready, some package is complete at an
interrupt or uninstall-interrupt stage, and no package is erroring. The
predicate is positive evidence of recovery rather than the absence of an error,
so failures outside a completed interrupt still mark the node `erroring` and
still stop the rollout: a drain timeout, any erroring package, an unreadable
nodeState annotation, and a node that is not Ready are all held. The new branch
is guarded by `!node.IsComplete()` so it cannot overwrite a `Complete` that the
preceding branch just wrote. `failureThreshold`, the batch success threshold and
exit-code handling are unchanged.

`SetStatus` now compares the status annotation and the status label to the
desired status independently, so a label that drifted on its own is repaired and
actually patched back. Previously the whole metadata write was gated on the
annotation differing, which made the label write dead code on the one path that
existed to fix it -- `IntrospectNode` calls `SetStatus(node.Status())`, and
`Status()` reads the annotation, so the comparison was always equal. `Status()`
still reads only the annotation: both reset commands delete the annotation and
the label in two separate API calls and only warn if the label call fails, so a
label surviving without its annotation is the wreckage of a failed reset and
must not resurrect a cleared status.

Recovering an already-wed... (continued)

45 of 60 new or added lines in 5 files covered. (75.0%)

9519 of 11609 relevant lines covered (82.0%)

7.62 hits per line