NVIDIA / k8s-driver-manager
12%
main: 11%

LAST BUILD BRANCH: pull-request/266
DEFAULT BRANCH: main
Repo Added
Token XZJCBg9TTAnxoAadYnqcc5m1o1EgNwco8 regen
Build 489 Last
Files 10
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

LAST BUILD ON BRANCH pull-request/263
branch: pull-request/263
CHANGE BRANCH
x
Sync Branches

coverage: 11.806% (+0.7%) from 11.072%
34541594029

Pull #263

github

rajathagasthya 
Invalidate status files before rescheduling operands

The operator-validator and driver containers remove the readiness files
under /run/nvidia/validations in their preStop hooks. Those hooks do not
run when the containers are removed outside the kubelet, for example
when a CRI-O minor version upgrade wipes every pod sandbox on the node.
The files survive on the host tmpfs. When k8s-driver-manager then
re-enables the operand labels, the operand pods are recreated together
with the driver pod, before the driver is installed. A stale
toolkit-ready lets an operand init container pass while the kernel
module is unloaded, so the nvidia runtime creates the main container
without the driver libraries. The MIG manager then fails NVML
initialization with ERROR_LIBRARY_NOT_FOUND and leaves the node at
mig.config.state=failed until the pod is deleted.

Remove the files right after the GPU operator components have been
evicted. The validators are confirmed deleted at that point, so nothing
recreates the files until they are rescheduled against the replacement
driver. The paths that do not evict, because no driver is loaded or
because eviction itself failed, leave the files alone: a validator that
is still running would not rewrite them.

The set removed is what the two preStop hooks remove: the *-ready files
written by the validators, plus the driver's .driver-ctr-ready and
.driver-daemons-status. Other hidden markers, such as cc-manager's
.cc-manager-ctr-ready, are left in place because k8s-driver-manager
does not restart their owners. The sandbox validator waits on that
marker before writing cc-manager-ready, and cc-manager can share a node
with the vfio-manager DaemonSet, which also runs this init container.
workload-type is not a readiness marker and is also left in place.

In a normal driver rollout both preStop hooks have already run and the
removal is a no-op. On the fast path that skips the module unload,
removing .driver-ctr-ready means the validator wai... (continued)
Pull Request #263: Invalidate status files before rescheduling operands

15 of 22 new or added lines in 1 file covered. (68.18%)

202 of 1711 relevant lines covered (11.81%)

0.13 hits per line

Relevant lines Covered
1711 RELEVANT LINES 202 COVERED LINES
0.13 HITS PER LINE
Source Files on pull-request/263

Recent builds

Builds Branch Commit Type Ran Committer Via Coverage
34541594029 pull-request/263 Invalidate status files before rescheduling operands The operator-validator and driver containers remove the readiness files under /run/nvidia/validations in their preStop hooks. Those hooks do not run when the containers are removed outside the ... Pull #263 rajathagasthya github
11.81
See All Builds (489)

Badge your Repo: k8s-driver-manager

We detected this repo isn’t badged! Grab the embed code to the right, add it to your repo to show off your code coverage, and when the badge is live hit the refresh button to remove this message.
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

Refresh