NVIDIA / gpu-operator / 28384664771
32%
main: 31%

LAST BUILD BRANCH: pull-request/2517
DEFAULT BRANCH: main
Ran
Jobs 1
Files 58
Run time 1min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 31.519% (-0.01%) from 31.529%
28384664771

Pull #2573

github

rajathagasthya 
Add Cluster Autoscaler integration example with NPD and NRC

When the Cluster Autoscaler adds a GPU node, the node reports Ready
long before the driver, container toolkit, and device plugin are
installed. Workloads scheduled in that window fail or occupy the node
so the autoscaler considers the scale-up satisfied, and MIG node pools
can deadlock entirely when scaling from zero.

Add a reference guide and example manifests that gate scheduling on
GPU readiness using upstream components: the node pool template
applies a startup taint, Node Problem Detector publishes a
nvidia.com/GPUReady node condition from an nvidia-smi probe, and the
Node Readiness Controller removes the taint once the condition is
True. The GPU Operator itself is unchanged; its operands only need
tolerations via the existing daemonsets.tolerations and NFD worker
toleration values.

The example includes a kind-based simulation for clusters without
GPUs: workers join pre-tainted via kubelet registration (matching
node pool template semantics), a marker file stands in for the
nvidia-smi probe, and the walkthrough downloads NRC's kindscaler.sh
to simulate a scale-up by adding a fresh worker to the running
cluster.

Known limitation documented in the guide: the Cluster Autoscaler must
be told to treat the readiness.k8s.io/ taint as a startup taint. This
works on a self-managed autoscaler today and is planned to work
natively on GKE; on AKS and other managed autoscalers it depends on
the provider presetting the prefix or exposing the flag.

Signed-off-by: Rajath Agasthya <ragasthya@nvidia.com>
Pull Request #2573: Add Cluster Autoscaler integration example with NPD and NRC

4025 of 12770 relevant lines covered (31.52%)

0.36 hits per line

Coverage Regressions

Lines Coverage ∆ File
55
0.0
 0.0% controllers/upgrade_controller.go
40
35.43
 -0.13% controllers/state_manager.go
25
0.0
 0.0% controllers/clusterpolicy_controller.go
11
0.0
 0.0% cmd/gpu-operator/main.go
Jobs
ID Job ID Ran Files Coverage
1 28384664771.1 58
31.52
 GitHub Action Run
Source Files on build 28384664771