NVIDIA / nodewright / 32907362950

82%

Build Type

push

github

Committed by web-flow

Commit Message

fix(operator): wait for evicted pods to terminate before interrupt (#539)

* fix(operator): wait for evicted pods to terminate before interrupt

DecidePod classified any pod carrying a deletionTimestamp as ignorable, and
BlocksDrain is defined as "not ignored", so a terminating pod neither got
re-evicted (correct) nor blocked drain (incorrect). IsDrained therefore returned
true as soon as every evictable pod merely *had* a deletionTimestamp: with the
2s requeue, pass 1 evicted, pass 2 saw everything terminating and reported the
node drained, and the interrupt fired. A pod with a 30s
terminationGracePeriodSeconds was ~2s into shutdown when the node rebooted.

Terminating pods now return ActionBlock. Both call sites already do the right
thing with that: IsDrained counts the pod as still present, and the DrainNode
switch maps ActionBlock to waitingForPods without re-evicting. This matches
kubectl drain, which waits for the pods it selected to disappear.

The check also moves below the exemptions. It previously sat second, above the
package-pod, unschedulable-toleration, DaemonSet, kube-system, and mirror-pod
filters, where flipping it in place would have made a terminating DaemonSet pod
mid-rollout - or one of the operator's own package pods - block drain, which is
a self-deadlock the exemptions exist to prevent. kubectl drain never waits on
pods it did not select either.

drainConfig.timeout consequently measures time-to-drain rather than
time-to-accept-evictions, which reads like the wait it was meant to bound.
Its semantics are unchanged: unset or zero is still no timeout, so a pod that
can never finish terminating holds the node in in_progress until it is cleared.
No default timeout is introduced here; that would newly mark PDB-gated drains
erroring, which is a separate decision from this fix.

Closes #534

Signed-off-by: Alex Yuskauskas <ayuskauskas@nvidia.com>

* docs(operator): correct drain timeout start and evict/delete wording

Two wording fixe... (continued)

Coverage Stats

3 of 3 new or added lines in 1 file covered. (100.0%)

11 existing lines in 2 files now uncovered.

9127 of 11080 relevant lines covered (82.37%)

7.64 hits per line