SchSeba / sriov-network-operator-1 / 26745654512 / 1
63%
master: 64%

LAST BUILD BRANCH: fix_drain_issue_v2
DEFAULT BRANCH: master
Ran
Files 108
Run time 4s
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 63.353% (-0.09%) from 63.443%
26745654512.1

push

github

SchSeba 
Fix drain escalation race that could reboot node without full drain

A race condition existed where a node could be rebooted with production
workloads still running. The scenario:

1. Daemon requests Drain_Required (partial drain - only SR-IOV pods evicted)
2. Drain controller performs partial drain and sets DrainComplete
3. Meanwhile, a new policy requiring reboot (e.g. RDMA mode change) arrives
4. Daemon sees DrainComplete and proceeds to reboot without a full drain

The root cause is that the daemon could not distinguish whether the
completed drain was sufficient for its current (escalated) requirements.

Fix: introduce a new annotation `sriovnetwork.openshift.io/drain-action`
on SriovNetworkNodeState, owned by the drain controller, that records the
drain type actually performed. The daemon verifies this matches its current
need before proceeding to apply configuration or reboot.

The drain controller:
- Sets drain-action when starting a drain (atomically with current-state)
- Re-reads desired-state after drain completes to detect mid-drain escalation
- Resets to Draining and re-drains if desired-state escalated beyond drain-action
- Clears drain-action when returning to Idle

The daemon:
- Checks drain-action satisfies its requirement at DrainComplete
- Escalates desired-state during Draining if reboot becomes needed
- Never proceeds to apply/reboot if drain-action is insufficient

Also adds DrainStateAnnotationPredicate watch on desired-state changes and
AnnotateObjectMultiple utility to batch annotation writes in a single API call.

Signed-off-by: Sebastian Sch <sebassch@gmail.com>

9539 of 15057 relevant lines covered (63.35%)

0.7 hits per line

Source Files on job 26745654512.1