36761064813
44%
main: 45%

Ran 30 Sep 2026 06:55PM UTC

Jobs 1

Files 75

Run time 1min

Badge

Embed ▾

Committed 30 Sep 2026 06:45PM UTC coverage: 44.494% (+6.6%) from 37.904%

Build # 36761064813

Build Type

push

github

Committed by

abrarshivani

Commit Message

ci: probe why the cluster became unreachable

A containerd run lost the API server twenty seconds after the driver pod
began reloading host kernel modules, and never got it back: 45 minutes of
retries, and the post-run diagnostics still could not connect.

Two very different causes fit that evidence equally well. Either the node
lost its network during the module reload, or the runner's egress address
changed and no longer matches the security group rule Holodeck opened for
it. The logs cannot separate them because nothing touches the node after
the kernel modules are loaded, so a silent API server is the only symptom
we ever see.

Record the egress address while things still work, then on every run
report it again alongside a direct readyz check, an ssh probe of the node
on port 22, and the instance state from EC2. Whichever of those still
answers tells us which side broke. It runs on always() so a passing run
leaves a baseline to compare a failing one against, and every probe is
bounded and swallowed so it cannot change the result of the job.

Signed-off-by: Abrar Shivani <ashivani@nvidia.com>

Coverage Stats

5770 of 12968 relevant lines covered (44.49%)

0.45 hits per line

Coverage Regressions

Lines	Coverage	∆	File
442	60.25	0.02%	controllers/object_controls.go
376	10.83	-0.9%	cmd/nvidia-validator/main.go
99	11.16	0.0%	api/nvidia/v1/clusterpolicy_types.go
77	33.65	1.07%	controllers/clusterpolicy_controller.go
19	12.69	-0.26%	cmd/gpu-operator/main.go
15	66.67	0.0%	internal/utils/utils.go
11	50.7	0.0%	controllers/state_manager.go
5	87.8	87.8%	internal/state/manager.go
2	95.06	3.7%	internal/state/nodepool.go

Jobs

ID	Job ID	Ran	Files	Coverage
1	36761064813.1	30 Sep 2026 06:55PM UTC	75	44.49	GitHub Action Run

NVIDIA / gpu-operator / 36761064813
44%
main: 45%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Coverage Regressions

Jobs

Source Files on build 36761064813

NVIDIA / gpu-operator / 36761064813 44% main: 45%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Coverage Regressions

Jobs

Source Files on build 36761064813

NVIDIA / gpu-operator / 36761064813
44%
main: 45%

README BADGES
x