NVIDIA / gpu-operator
32%
main: 31%

LAST BUILD BRANCH: pull-request/2517
DEFAULT BRANCH: main
Repo Added
Files 56
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

LAST BUILD ON BRANCH pull-request/2573
branch: pull-request/2573
CHANGE BRANCH
x

coverage: 31.519% (-0.01%) from 31.529%
28384664771

Pull #2573

github

rajathagasthya 
Add Cluster Autoscaler integration example with NPD and NRC

When the Cluster Autoscaler adds a GPU node, the node reports Ready
long before the driver, container toolkit, and device plugin are
installed. Workloads scheduled in that window fail or occupy the node
so the autoscaler considers the scale-up satisfied, and MIG node pools
can deadlock entirely when scaling from zero.

Add a reference guide and example manifests that gate scheduling on
GPU readiness using upstream components: the node pool template
applies a startup taint, Node Problem Detector publishes a
nvidia.com/GPUReady node condition from an nvidia-smi probe, and the
Node Readiness Controller removes the taint once the condition is
True. The GPU Operator itself is unchanged; its operands only need
tolerations via the existing daemonsets.tolerations and NFD worker
toleration values.

The example includes a kind-based simulation for clusters without
GPUs: workers join pre-tainted via kubelet registration (matching
node pool template semantics), a marker file stands in for the
nvidia-smi probe, and the walkthrough downloads NRC's kindscaler.sh
to simulate a scale-up by adding a fresh worker to the running
cluster.

Known limitation documented in the guide: the Cluster Autoscaler must
be told to treat the readiness.k8s.io/ taint as a startup taint. This
works on a self-managed autoscaler today and is planned to work
natively on GKE; on AKS and other managed autoscalers it depends on
the provider presetting the prefix or exposing the flag.

Signed-off-by: Rajath Agasthya <ragasthya@nvidia.com>
Pull Request #2573: Add Cluster Autoscaler integration example with NPD and NRC

4025 of 12770 relevant lines covered (31.52%)

0.36 hits per line

Relevant lines Covered
12770 RELEVANT LINES 4025 COVERED LINES
0.36 HITS PER LINE
Source Files on pull-request/2573

Recent builds

Builds Branch Commit Type Ran Committer Via Coverage
28384664771 pull-request/2573 Add Cluster Autoscaler integration example with NPD and NRC When the Cluster Autoscaler adds a GPU node, the node reports Ready long before the driver, container toolkit, and device plugin are installed. Workloads scheduled in that window fail or... Pull #2573 rajathagasthya github
31.52
28039030267 pull-request/2573 Add Cluster Autoscaler integration example with NPD and NRC When the Cluster Autoscaler adds a GPU node, the node reports Ready long before the driver, container toolkit, and device plugin are installed. Workloads scheduled in that window fail or... Pull #2573 rajathagasthya github
31.52
See All Builds (1953)