NVIDIA / k8s-driver-manager / 39866926469

22%

main: 13%

Build Type

Pull #291

github

Committed by rajathagasthya

Commit Message

Skip GPU claim-holders that wait for a missing DRA kubelet-plugin

When an upgrade restarts the driver pod while no DRA kubelet-plugin pod
is Ready, GPU Operator pods that hold GPU claims cannot finish
terminating. The kubelet needs the kubelet-plugin to unprepare their
claims, and a new kubelet-plugin pod starts only after the driver is
ready. The driver-manager waited for these pods until it timed out, so
the driver pod never became ready. Stop waiting for a GPU Operator pod
when its containers have stopped, it holds a GPU claim, and no
kubelet-plugin pod is Ready. The kubelet unprepares the claims after
the kubelet-plugin starts again, unless gpu.deploy.dra-driver is false,
which gets its own warning. When the GPU Operator upgrade policy is
off, the driver-manager evicts GPU pods itself. That step also leaves
these pods out, because it cannot delete DaemonSet pods and fails when
it counts them. In the full path, also wait for ComputeDomain IMEX
daemon pods before the kubelet-plugin is drained: they use all GPUs
through a CDI spec without a GPU claim, and the driver unload fails
while they run. A terminating daemon pod whose containers have stopped
is not waited for. User pods that hold GPU claims still stop the driver
unload.

Signed-off-by: Rajath Agasthya <ragasthya@nvidia.com>

Pull Request Pull Request #291: Skip GPU claim-holders that wait for a missing DRA kubelet-plugin

Coverage Stats

84 of 145 new or added lines in 2 files covered. (57.93%)

1 existing line in 1 file now uncovered.

374 of 1678 relevant lines covered (22.29%)

0.22 hits per line