SchSeba / sriov-network-operator-1 / 26745654512

push

github

Fix drain escalation race that could reboot node without full drain

A race condition existed where a node could be rebooted with production
workloads still running. The scenario:

1. Daemon requests Drain_Required (partial drain - only SR-IOV pods evicted)
2. Drain controller performs partial drain and sets DrainComplete
3. Meanwhile, a new policy requiring reboot (e.g. RDMA mode change) arrives
4. Daemon sees DrainComplete and proceeds to reboot without a full drain

The root cause is that the daemon could not distinguish whether the
completed drain was sufficient for its current (escalated) requirements.

Fix: introduce a new annotation `sriovnetwork.openshift.io/drain-action`
on SriovNetworkNodeState, owned by the drain controller, that records the
drain type actually performed. The daemon verifies this matches its current
need before proceeding to apply configuration or reboot.

The drain controller:
- Sets drain-action when starting a drain (atomically with current-state)
- Re-reads desired-state after drain completes to detect mid-drain escalation
- Resets to Draining and re-drains if desired-state escalated beyond drain-action
- Clears drain-action when returning to Idle

The daemon:
- Checks drain-action satisfies its requirement at DrainComplete
- Escalates desired-state during Draining if reboot becomes needed
- Never proceeds to apply/reboot if drain-action is insufficient

Also adds DrainStateAnnotationPredicate watch on desired-state changes and
AnnotateObjectMultiple utility to batch annotation writes in a single API call.

Signed-off-by: Sebastian Sch <sebassch@gmail.com>

121 of 167 new or added lines in 6 files covered. (72.46%)

9539 of 15057 relevant lines covered (63.35%)

0.7 hits per line