kubeflow / trainer
58%
master: 58%

LAST BUILD BRANCH: feat/trainer-multi-slice-tpu
DEFAULT BRANCH: master
Repo Added
Token 3qIdUH6ns6RNy0sBPPQ6ybJp7VqYkScU8 regen
Build 3006 Last
Files 40
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

LAST BUILD ON BRANCH feat/trainer-multi-slice-tpu
branch: SELECT
CHANGE BRANCH
x
Sync Branches

coverage: 57.792% (-0.3%) from 58.057%
23960834318

Pull #3408

github

krishdef7 
feat(operator): support multi-slice TPU training via trainer replicas

For multi-slice TPU, JobSet models each TPU slice as a ReplicatedJob
replica, with parallelism = hosts per slice and replicas = slice count.
The operator previously blocked this with two hard constraints:

1. builder.go unconditionally set trainer Replicas = 1, destroying any
   value from the runtime template.
2. trainingruntime_webhook.go rejected replicas != 1 for all ancestors
   including trainer.

Changes:
- builder.go: nil-guard for trainer Replicas, preserving the value from
  the runtime template instead of unconditional overwrite.
- jobset.go: in Build(), compute perSlice = numNodes / replicas for the
  trainer ancestor so each slice runs the correct number of hosts.
- trainingruntime_webhook.go: allow trainer ancestor replicas > 1 to
  enable multi-slice configurations to pass admission.
- trainingruntime_webhook_test.go: update invalid_replicas test to
  reflect that trainer replicas > 1 is now valid.
- trainingruntime_test.go: add test case for 4-slice x 8 hosts
  (NumNodes=32), verifying Parallelism=8 per slice and MinMember=34.

Semantics: numNodes = total hosts across all slices.
Per-slice hosts = numNodes / replicas.

REF: https://github.com/kubeflow/trainer/issues/3407
Signed-off-by: krishdef7 <gargkrish06@gmail.com>
Pull Request #3408: feat(operator): support multi-slice TPU by enabling trainer replicas > 1

6 of 32 new or added lines in 4 files covered. (18.75%)

2036 of 3523 relevant lines covered (57.79%)

0.67 hits per line

Relevant lines Covered
3523 RELEVANT LINES 2036 COVERED LINES
0.67 HITS PER LINE
Source Files on master

Recent builds

Builds Branch Commit Type Ran Committer Via Coverage
23960834318 feat/trainer-multi-slice-tpu feat(operator): support multi-slice TPU training via trainer replicas For multi-slice TPU, JobSet models each TPU slice as a ReplicatedJob replica, with parallelism = hosts per slice and replicas = slice count. The operator previously blocked thi... Pull #3408 krishdef7 github
57.79
23960692171 feat/trainer-multi-slice-tpu feat(operator): support multi-slice TPU training via trainer replicas For multi-slice TPU, JobSet models each TPU slice as a ReplicatedJob replica, with parallelism = hosts per slice and replicas = slice count. The operator previously blocked thi... Pull #3408 krishdef7 github
57.89
23811025952 test-statusserver-helpers fix(statusserver): improve bearer token parsing and add helper tests Signed-off-by: Skolli <tanusuch@gmail.com> Pull #3405 suchirkolli github
58.34
23957046905 dependabot/docker/cmd/trainers/torchtune/pytorch/pytorch-2.11.0-cuda12.8-cudnn9-runtime chore(deps): bump pytorch/pytorch in /cmd/trainers/torchtune Bumps pytorch/pytorch from 2.9.1-cuda12.8-cudnn9-runtime to 2.11.0-cuda12.8-cudnn9-runtime. --- updated-dependencies: - dependency-name: pytorch/pytorch dependency-version: 2.11.0-cu... Pull #3381 web-flow github
58.06
23957029240 dependabot/docker/cmd/runtimes/deepspeed/nvidia/cuda-13.2.0-devel-ubuntu22.04 chore(deps): bump nvidia/cuda in /cmd/runtimes/deepspeed Bumps nvidia/cuda from 13.1.1-devel-ubuntu22.04 to 13.2.0-devel-ubuntu22.04. --- updated-dependencies: - dependency-name: nvidia/cuda dependency-version: 13.2.0-devel-ubuntu22.04 depen... Pull #3380 web-flow github
58.14
23956912610 dependabot/pip/cmd/runtimes/deepspeed/deepspeed-0.18.9 chore(deps): bump deepspeed in /cmd/runtimes/deepspeed Bumps [deepspeed](https://github.com/deepspeedai/DeepSpeed) from 0.18.7 to 0.18.9. - [Release notes](https://github.com/deepspeedai/DeepSpeed/releases) - [Commits](https://github.com/deepspee... Pull #3402 web-flow github
58.06
23956913849 dependabot/pip/cmd/runtimes/deepspeed/deepspeed-0.18.9 chore(deps): bump deepspeed in /cmd/runtimes/deepspeed Bumps [deepspeed](https://github.com/deepspeedai/DeepSpeed) from 0.18.7 to 0.18.9. - [Release notes](https://github.com/deepspeedai/DeepSpeed/releases) - [Commits](https://github.com/deepspee... Pull #3402 web-flow github
58.06
23956914515 dependabot/pip/cmd/runtimes/deepspeed/datasets-4.8.4 chore(deps): bump datasets in /cmd/runtimes/deepspeed Bumps [datasets](https://github.com/huggingface/datasets) from 4.7.0 to 4.8.4. - [Release notes](https://github.com/huggingface/datasets/releases) - [Commits](https://github.com/huggingface/da... Pull #3384 web-flow github
58.06
23956910435 master chore(deps): bump github.com/go-jose/go-jose/v4 from 4.1.3 to 4.1.4 (#3406) Bumps [github.com/go-jose/go-jose/v4](https://github.com/go-jose/go-jose) from 4.1.3 to 4.1.4. - [Release notes](https://github.com/go-jose/go-jose/releases) - [Commits](... push web-flow github
58.06
23956878291 master chore(deps): bump transformers from 5.3.0 to 5.4.0 in /cmd/runtimes/deepspeed (#3401) Bumps [transformers](https://github.com/huggingface/transformers) from 5.3.0 to 5.4.0. - [Release notes](https://github.com/huggingface/transformers/releases) -... push web-flow github
58.14
See All Builds (2687)

Badge your Repo: trainer

We detected this repo isn’t badged! Grab the embed code to the right, add it to your repo to show off your code coverage, and when the badge is live hit the refresh button to remove this message.
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

Refresh