SchSeba / sriov-network-operator-1 / 26741408493

63%

master: 64%

Build Type

push

github

Committed by SchSeba

Commit Message

Fix drain escalation race that could reboot node without full drain

A race condition existed where a node could be rebooted with production
workloads still running. The scenario:

1. Daemon requests Drain_Required (partial drain - only SR-IOV pods evicted)
2. Drain controller performs partial drain and sets DrainComplete
3. Meanwhile, a new policy requiring reboot (e.g. RDMA mode change) arrives
4. Daemon sees DrainComplete and proceeds to reboot without a full drain

The root cause is that the daemon could not distinguish whether the
completed drain was sufficient for its current (escalated) requirements.

Fix: introduce a new annotation `sriovnetwork.openshift.io/drain-action`
on SriovNetworkNodeState, owned by the drain controller, that records the
drain type actually performed. The daemon verifies this matches its current
need before proceeding to apply configuration or reboot.

The drain controller:
- Sets drain-action when starting a drain (atomically with current-state)
- Re-reads desired-state after drain completes to detect mid-drain escalation
- Resets to Draining and re-drains if desired-state escalated beyond drain-action
- Clears drain-action when returning to Idle

The daemon:
- Checks drain-action satisfies its requirement at DrainComplete
- Escalates desired-state during Draining if reboot becomes needed
- Never proceeds to apply/reboot if drain-action is insufficient

Also adds DrainStateAnnotationPredicate watch on desired-state changes and
AnnotateObjectMultiple utility to batch annotation writes in a single API call.

Signed-off-by: Sebastian Sch <sebassch@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

Coverage Stats

115 of 152 new or added lines in 6 files covered. (75.66%)

24 existing lines in 5 files now uncovered.

9543 of 15052 relevant lines covered (63.4%)

0.71 hits per line