NVIDIA / nvidia-container-toolkit
37%
main: 38%

LAST BUILD BRANCH: pull-request/1444
DEFAULT BRANCH: main
Repo Added
Token yURQA9qJZkSTLsagBnXzxNupHtB5LQLk7 regen
Build 1361 Last
Files 230
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

LAST BUILD ON BRANCH pull-request/1400
branch: pull-request/1400
CHANGE BRANCH
x
Sync Branches

coverage: 37.283%. Remained the same
18958090005

push

github

cdesiniotis 
Redirect log message to stderr in nvidia runtime wrapper script

This change is required to make our nvidia runtime wrapper compliant with
the OCI runtime spec. All OCI-compliant runtimes must support the operations
documented at https://github.com/opencontainers/runtime-spec/blob/v1.2.1/runtime.md#operations.
Before this change, our nvidia runtime wrapper was not producing the expected
output when the query state operation (`state <container-id>`) was invoked
AND the nvidia kernel modules happened to not be loaded. In this case, we were
emitting an extra log message which caused the stdout of this command to not
adhere to the schema defined in the OCI runtime spec. Redirecting the log
message to stderr makes us compliant.

This issue was discovered when deploying GPU Operator 25.10.0 on nodes using cri-o.
GPU Operator 25.10.0 is the first release that installs nvidia runtime handlers
with cri-o by default, as opposed to installing an OCI hook file. When performing
a GPU driver upgrade, pods in the gpu-operator namespace would be in the
`Init:RunContainerError` state for several minutes until the new driver finished
installing -- note that no nvidia driver modules are loaded during this span of
several minutes. When inspecting the cri-o logs, we observed the following error
message:

```
level=warning msg="Error updating the container status \"<a class=hub.com/NVIDIA/nvidia-container-toolkit/commit/16779f4cd2414a164aae56856b491f86fe0c6b80">16779f4cd3a3b4474ada2cc0864c8e028\": failed to decode container status for 16779f4cd2414a164aae56856b491f86fe0c6b803a3b4474ada2cc0864c8e028: skipThreeBytes: expect ull, error found in #2 byte of ...|nvidia drive|..., bigger context ...|nvidia driver modules are not yet loaded, invoking /|..." id=a4b48041-edc4-48c2-8d75-4ad03cb3d8e1 name=/runtime.v1.RuntimeService/CreateContainer
```

This error message indicates cri-o failed to get the status of the container because
it could not decode the JSON returned by the runtime handler.

Signed-off-by: Christopher Desiniotis <cdesiniotis@nvidia.com>

1 of 1 new or added line in 1 file covered. (100.0%)

5037 of 13510 relevant lines covered (37.28%)

0.42 hits per line

Relevant lines Covered
13510 RELEVANT LINES 5037 COVERED LINES
0.42 HITS PER LINE
Source Files on pull-request/1400

Recent builds

Builds Branch Commit Type Ran Committer Via Coverage
18958090005 pull-request/1400 Redirect log message to stderr in nvidia runtime wrapper script This change is required to make our nvidia runtime wrapper compliant with the OCI runtime spec. All OCI-compliant runtimes must support the operations documented at https://github.co... push cdesiniotis github
37.28
18957292195 pull-request/1400 Redirect log message to stderr in nvidia runtime wrapper script This change is required to make our nvidia runtime wrapper compliant with the OCI runtime spec. All OCI-compliant runtimes must support the operations documented at https://github.co... push cdesiniotis github
37.28
See All Builds (1318)

Badge your Repo: nvidia-container-toolkit

We detected this repo isn’t badged! Grab the embed code to the right, add it to your repo to show off your code coverage, and when the badge is live hit the refresh button to remove this message.
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

Refresh