NVIDIA / k8s-driver-manager / 39480345494

22%

main: 13%

Build Type

Pull #291

github

Committed by rajathagasthya

Commit Message

Skip GPU claim-holders that wait for a missing DRA kubelet-plugin

When an upgrade restarts the driver pod while no DRA kubelet-plugin pod
is Ready, GPU Operator pods that hold GPU claims cannot finish
terminating. The kubelet needs the kubelet-plugin to unprepare their
claims, and a new kubelet-plugin pod starts only after the driver is
ready. The driver-manager waited for these pods until it timed out, so
the driver pod never became ready. Stop waiting for a GPU Operator pod
when its containers have stopped, it holds a GPU claim, and no
kubelet-plugin pod is Ready. The kubelet unprepares the claims after
the kubelet-plugin starts again, unless gpu.deploy.dra-driver is false,
which gets its own warning. When the GPU Operator upgrade policy is
off, the driver-manager evicts GPU pods itself. That step also leaves
these pods out, because it cannot delete DaemonSet pods and fails when
it counts them. User pods that hold GPU claims still stop the driver
unload.

Signed-off-by: Rajath Agasthya <ragasthya@nvidia.com>

Pull Request Pull Request #291: Skip GPU claim-holders that wait for a missing DRA kubelet-plugin

Coverage Stats

40 of 92 new or added lines in 2 files covered. (43.48%)

1 existing line in 1 file now uncovered.

331 of 1626 relevant lines covered (20.36%)

0.2 hits per line