NVIDIA / gpu-operator / 23916641071
28%
main: 28%

LAST BUILD BRANCH: pull-request/2250
DEFAULT BRANCH: main
Ran
Jobs 1
Files 54
Run time 1min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 27.711%. Remained the same
23916641071

Pull #2264

github

rajathagasthya 
Prevent operand pods from starting before nvidia driver is loaded

When an operand pod (mig-manager, device-plugin, GFD, etc.) is created
while the nvidia driver module is not loaded, the nvidia container runtime
wrapper falls back to plain runc/crun. This means no NVIDIA libraries are
injected into the container — the container is permanently broken for its
lifetime.

The init containers in these pods gate on the toolkit-ready sentinel file,
but this file can be validly present from a running operator-validator
even while the driver is temporarily unloaded (e.g., during a driver
container restart). The sentinel file system validates that components
were installed correctly but does not validate that the nvidia kernel
module is currently loaded — which is the actual condition required for
the container runtime to inject NVIDIA libraries.

Add a /proc/modules check for the nvidia kernel module alongside the
existing toolkit-ready file check in all operand init containers. This
ensures the main container is not created until the nvidia module is
loaded, guaranteeing the runtime wrapper invokes the real
nvidia-container-runtime (which injects libraries) instead of falling
back to runc.

Additionally, add preStop lifecycle hooks to the toolkit and driver
containers to clean up sentinel files (toolkit-ready, driver-ready) on
graceful termination. This prevents downstream init containers from
passing on stale files when pods are properly terminated during upgrades
or drains.

Co-authored-by: Shiva Kumar <shivaku@nvidia.com>
Signed-off-by: Rajath Agasthya <ragasthya@nvidia.com>
Pull Request #2264: Prevent operand pods from starting before nvidia driver is loaded

3381 of 12201 relevant lines covered (27.71%)

0.32 hits per line

Jobs
ID Job ID Ran Files Coverage
1 23916641071.1 54
27.71
 GitHub Action Run
Source Files on build 23916641071