SchSeba / sriov-network-operator-1
63%
master: 64%

LAST BUILD BRANCH: devtask-d6444829-add-corerabbit-yaml-configuration-for-th
DEFAULT BRANCH: master
Repo Added
Token MCzExkrUqCmca7JovFWU5quDblYDZ2Jzk regen
Build 803 Last
Files 108
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

LAST BUILD ON BRANCH fix_drain_issue
branch: fix_drain_issue
CHANGE BRANCH
x
Sync Branches

coverage: 63.473% (+0.03%) from 63.443%
26826204253

push

github

SchSeba 
Fix drain escalation race that could reboot node without full drain

A race condition existed where a node could be rebooted with production
workloads still running. The scenario:

1. Daemon requests Drain_Required (partial drain - only SR-IOV pods evicted)
2. Drain controller performs partial drain and sets DrainComplete
3. Meanwhile, a new policy requiring reboot (e.g. RDMA mode change) arrives
4. Daemon sees DrainComplete and proceeds to reboot without a full drain

The root cause is that the daemon could not distinguish whether the
completed drain was sufficient for its current (escalated) requirements.

Fix: introduce a new annotation `sriovnetwork.openshift.io/drain-action`
on SriovNetworkNodeState, owned by the drain controller, that records the
drain type actually performed. The daemon verifies this matches its current
need before proceeding to apply configuration or reboot.

The drain controller:
- Sets drain-action when starting a drain (atomically with current-state)
- Re-reads desired-state after drain completes to detect mid-drain escalation
- Resets to Draining and re-drains if desired-state escalated beyond drain-action
- Clears drain-action when returning to Idle

The daemon:
- Checks drain-action satisfies its requirement at DrainComplete
- Escalates desired-state during Draining if reboot becomes needed
- Never proceeds to apply/reboot if drain-action is insufficient

Also adds DrainStateAnnotationPredicate watch on desired-state changes and
AnnotateObjectMultiple utility to batch annotation writes in a single API call.

Signed-off-by: Sebastian Sch <sebassch@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Sebastian Sch <sebassch@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

128 of 174 new or added lines in 6 files covered. (73.56%)

24 existing lines in 5 files now uncovered.

9561 of 15063 relevant lines covered (63.47%)

0.71 hits per line

Relevant lines Covered
15063 RELEVANT LINES 9561 COVERED LINES
0.71 HITS PER LINE
Source Files on fix_drain_issue

Recent builds

Builds Branch Commit Type Ran Committer Via Coverage
26826204253 fix_drain_issue Fix drain escalation race that could reboot node without full drain A race condition existed where a node could be rebooted with production workloads still running. The scenario: 1. Daemon requests Drain_Required (partial drain - only SR-IOV pod... push SchSeba github
63.47
26745654512 fix_drain_issue Fix drain escalation race that could reboot node without full drain A race condition existed where a node could be rebooted with production workloads still running. The scenario: 1. Daemon requests Drain_Required (partial drain - only SR-IOV pod... push SchSeba github
63.35
26741474555 fix_drain_issue Fix drain escalation race that could reboot node without full drain A race condition existed where a node could be rebooted with production workloads still running. The scenario: 1. Daemon requests Drain_Required (partial drain - only SR-IOV pod... push SchSeba github
63.57
26741408493 fix_drain_issue Fix drain escalation race that could reboot node without full drain A race condition existed where a node could be rebooted with production workloads still running. The scenario: 1. Daemon requests Drain_Required (partial drain - only SR-IOV pod... push SchSeba github
63.4
See All Builds (691)

Badge your Repo: sriov-network-operator-1

We detected this repo isn’t badged! Grab the embed code to the right, add it to your repo to show off your code coverage, and when the badge is live hit the refresh button to remove this message.
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

Refresh