NVIDIA / k8s-driver-manager

12%

main: 11%

LAST BUILD ON BRANCH pull-request/263

branch: SELECT

CHANGE BRANCH

x

Sync Branches

• No branch selected
• copy-pr-bot
• custom-go-proxy
• dependabot/docker/deployments/container/nvidia/cuda-13.0.0-base-ubi9
• dependabot/docker/deployments/container/nvidia/cuda-13.0.1-base-ubi9
• dependabot/docker/deployments/container/nvidia/distroless/cc-v3.1.12-dev
• dependabot/docker/deployments/devel/golang-1.24.6
• dependabot/docker/deployments/devel/golang-1.25.1
• dependabot/github_actions/actions/checkout-5
• dependabot/github_actions/actions/setup-go-6
• dependabot/go_modules/main/github.com/NVIDIA/go-nvml-0.13.0-1
• dependabot/go_modules/main/golang.org/x/sys-0.35.0
• dependabot/go_modules/main/golang.org/x/sys-0.36.0
• dependabot/go_modules/main/k8sio-8aef8bcde0
• dm-go-rewrite
• fix-broken-ci
• fix-image-build-2
• local_devel_image
• main
• ngc_pipeline_changes
• pull-request/102
• pull-request/103
• pull-request/104
• pull-request/107
• pull-request/108
• pull-request/109
• pull-request/110
• pull-request/112
• pull-request/113
• pull-request/114
• pull-request/115
• pull-request/116
• pull-request/117
• pull-request/120
• pull-request/121
• pull-request/122
• pull-request/125
• pull-request/126
• pull-request/127
• pull-request/128
• pull-request/129
• pull-request/131
• pull-request/132
• pull-request/133
• pull-request/134
• pull-request/135
• pull-request/136
• pull-request/137
• pull-request/139
• pull-request/140
• pull-request/141
• pull-request/142
• pull-request/143
• pull-request/146
• pull-request/148
• pull-request/149
• pull-request/150
• pull-request/152
• pull-request/153
• pull-request/155
• pull-request/156
• pull-request/159
• pull-request/163
• pull-request/164
• pull-request/166
• pull-request/167
• pull-request/168
• pull-request/169
• pull-request/170
• pull-request/171
• pull-request/172
• pull-request/173
• pull-request/174
• pull-request/175
• pull-request/176
• pull-request/177
• pull-request/178
• pull-request/180
• pull-request/181
• pull-request/182
• pull-request/183
• pull-request/184
• pull-request/185
• pull-request/186
• pull-request/187
• pull-request/188
• pull-request/189
• pull-request/193
• pull-request/194
• pull-request/195
• pull-request/196
• pull-request/197
• pull-request/198
• pull-request/199
• pull-request/200
• pull-request/201
• pull-request/202
• pull-request/203
• pull-request/204
• pull-request/205
• pull-request/210
• pull-request/211
• pull-request/214
• pull-request/215
• pull-request/217
• pull-request/218
• pull-request/219
• pull-request/220
• pull-request/221
• pull-request/222
• pull-request/223
• pull-request/225
• pull-request/226
• pull-request/227
• pull-request/228
• pull-request/229
• pull-request/230
• pull-request/231
• pull-request/232
• pull-request/233
• pull-request/234
• pull-request/235
• pull-request/236
• pull-request/237
• pull-request/238
• pull-request/239
• pull-request/241
• pull-request/244
• pull-request/245
• pull-request/246
• pull-request/247
• pull-request/248
• pull-request/249
• pull-request/254
• pull-request/256
• pull-request/258
• pull-request/260
• pull-request/263
• release-0.10
• release-0.11
• release-0.12
• release-0.9

Build # 34541594029

Build Type

Pull #263

github

Committed by rajathagasthya

Commit Message

Invalidate status files before rescheduling operands

The operator-validator and driver containers remove the readiness files
under /run/nvidia/validations in their preStop hooks. Those hooks do not
run when the containers are removed outside the kubelet, for example
when a CRI-O minor version upgrade wipes every pod sandbox on the node.
The files survive on the host tmpfs. When k8s-driver-manager then
re-enables the operand labels, the operand pods are recreated together
with the driver pod, before the driver is installed. A stale
toolkit-ready lets an operand init container pass while the kernel
module is unloaded, so the nvidia runtime creates the main container
without the driver libraries. The MIG manager then fails NVML
initialization with ERROR_LIBRARY_NOT_FOUND and leaves the node at
mig.config.state=failed until the pod is deleted.

Remove the files right after the GPU operator components have been
evicted. The validators are confirmed deleted at that point, so nothing
recreates the files until they are rescheduled against the replacement
driver. The paths that do not evict, because no driver is loaded or
because eviction itself failed, leave the files alone: a validator that
is still running would not rewrite them.

The set removed is what the two preStop hooks remove: the *-ready files
written by the validators, plus the driver's .driver-ctr-ready and
.driver-daemons-status. Other hidden markers, such as cc-manager's
.cc-manager-ctr-ready, are left in place because k8s-driver-manager
does not restart their owners. The sandbox validator waits on that
marker before writing cc-manager-ready, and cc-manager can share a node
with the vfio-manager DaemonSet, which also runs this init container.
workload-type is not a readiness marker and is also left in place.

In a normal driver rollout both preStop hooks have already run and the
removal is a no-op. On the fast path that skips the module unload,
removing .driver-ctr-ready means the validator wai... (continued)

Pull Request Pull Request #263: Invalidate status files before rescheduling operands

Coverage Stats

15 of 22 new or added lines in 1 file covered. (68.18%)

202 of 1711 relevant lines covered (11.81%)

0.13 hits per line