NVIDIA / gpu-operator / 31064334768
38%
main: 38%

LAST BUILD BRANCH: pull-request/2753
DEFAULT BRANCH: main
Ran
Jobs 1
Files 73
Run time 1min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 38.385%. Remained the same
31064334768

push

github

abrarshivani 
fix(ci): survive a transient API outage during the driver upgrade

Both e2e jobs died in wait_for_driver_upgrade_done. The driver upgrade
restarts the container runtime on the node, and now that the tests drive the
cluster from the runner rather than from the node itself, the API server is
briefly unreachable across the public address instead of on localhost. The
first casualty was the opening kubectl in print_driver_upgrade_debug, which
unlike its three siblings had no guard, so a debug dump ended the run.

Guard it, and make the wait itself tolerate the outage: the node count and the
per-node upgrade state are now read through checks that treat a failure as
"not upgraded yet" and retry until the existing wall-clock deadline expires.
A count that could not be read stays empty rather than defaulting to zero, so
an unreachable API can never be mistaken for a finished upgrade. Failures are
announced with a timestamp so the next run shows how long such an outage
lasts, which this one died too quickly to reveal.

The calls in these two functions also carry an explicit request timeout, since
the default behaviour was to spend thirty seconds per call discovering that
the address was black-holed.

The same shape exists elsewhere: a kubectl whose output only exists for a
human to read, usually just before exit 1, sitting unguarded next to siblings
that already end in || true. Guard those too, in the readiness and log
collection loops in checks.sh and in the timeout dumps in
update-clusterpolicy.sh, migrate-clusterpolicy-to-nvidiadriver.sh and
update-nvidiadriver.sh. Assertions are left alone: a check that cannot reach
the API still fails the test.

Signed-off-by: Abrar Shivani <ashivani@nvidia.com>

5572 of 14516 relevant lines covered (38.39%)

0.44 hits per line

Jobs
ID Job ID Ran Files Coverage
1 31064334768.1 73
38.39
 GitHub Action Run
Source Files on build 31064334768