kubeflow / trainer / 30234662796
68%
master: 68%

LAST BUILD BRANCH: dependabot/cargo/pkg/data_cache/quinn-proto-0.11.16
DEFAULT BRANCH: master
Ran
Jobs 1
Files 42
Run time 1min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 68.256%. Remained the same
30234662796

Pull #3783

github

sanskar-singh-2403 
fix(runtimes): wait for launcher DNS before mpirun in MPI runtimes

The MLX and DeepSpeed ClusterTrainingRuntimes are MPI-based. #3748 made
the launcher depend on the node replicated job reaching Ready, so the
workers pass their sshd readiness probe and get their DNS records
published before the launcher starts. The launcher then runs mpirun
immediately, and orted on each worker connects back to the launcher
FQDN (<jobset>-launcher-0-0.<jobset>). That record is created later than
the workers' and can lag behind mpirun startup, so orted cannot route
back and the launcher exits with "ORTE was unable to reliably start one
or more daemons". launcher-to-worker resolution already works; only the
worker-to-launcher direction races.

Add a wait-for-launcher-dns initContainer to the launcher pod that
blocks until the launcher's own FQDN resolves via getent hosts before
the main container runs mpirun. This guarantees bidirectional DNS
resolution without the -mca routed direct workaround, which would paper
over the missing record rather than wait for it. The subdomain equals
the JobSet name, injected via the downward API from the
jobset.sigs.k8s.io/jobset-name label.

Applied to both the kustomize base manifests (used by the GPU E2E job)
and the Helm chart runtime files to keep them in sync.

Signed-off-by: Sanskar Singh <sanskarsinghty1234@gmail.com>
Pull Request #3783: chore: add MPI E2E tests for DeepSpeed and MLX example notebooks

2563 of 3755 relevant lines covered (68.26%)

0.8 hits per line

Jobs
ID Job ID Ran Files Coverage
1 30234662796.1 42
68.26
 GitHub Action Run
Source Files on build 30234662796