kubeflow / trainer

56%

master: 56%

LAST BUILD ON BRANCH trainjob-progress

branch: trainjob-progress

CHANGE BRANCH

x

Reset

Sync Branches

• trainjob-progress
• 2836-expose-builruntimeinfo
• 2871-allow-podspecoverride-dupl-jobs
• Bug
• KEP-volcano-scheduler
• a10-2-gpu
• add-akshay-reviewer
• add-audio-examples
• add-config-api-tests-2885
• add-dependabot
• add-gitattr
• add-license-scan-badge
• add-local-example
• add-local-trainer-client
• add-local-trainer-example
• add-manager-field-podtemplateoverride
• add-ok-to-test
• add-overlay-manifest-v2
• add-patch-updates-k8s
• add-pod-network-plugin-to-diagram
• add-qwen3-1.7b
• add-runtime-labels
• add-sdk-release
• add-standalone-manifest
• agents-md
• automate-release
• bo/feat/remove-launcher-chainer-validation
• bo/test/add-ut-for-torch-runtime-valid
• bump-go-1.25
• bump-jobset-v0.9.0
• bump-torch-deepspeed
• cache-example
• cache-oss
• cache_initilizer
• cache_pipeline
• changelog-1.9.1
• changelog-2.0.0
• changelog-2.0.1
• changelog-v2.0.0-rc.0
• changelog-v2.0.0-rc.1
• changelog-v2.1.0
• changelog-v2.1.0-rc.0
• changelog-v2.1.0-rc.1
• cherry-pick-2666-to-release-2.0
• cherry-pick-2675-to-release-2.0
• cherry-pick-2682-to-release-2.0
• cherry-pick-2683-to-release-2.0
• cherry-pick-2685-to-release-2.0
• cherry-pick-2686-to-release-2.0
• cherry-pick-2691-to-release-2.0
• cherry-pick-2695-to-release-2.0
• cherry-pick-2700-to-release-2.0
• cherry-pick-2703-to-release-2.0
• cherry-pick-2707-to-release-2.0
• cherry-pick-2719-to-release-2.0
• cherry-pick-2726-to-release-2.0
• cherry-pick-2728-to-release-2.1
• cherry-pick-2731-to-release-2.0
• cherry-pick-2734-to-release-2.0
• cherry-pick-2739-to-release-2.0
• cherry-pick-2761
• cherry-pick-2766
• cherry-pick-2771-to-release-2.0
• cherry-pick-2774-to-release-2.0
• cherry-pick-2780
• cherry-pick-2813
• cherry-pick-2815
• cherry-pick-2837-to-release-2.0
• cherry-pick-2854-to-release-2.0
• cherry-pick-2877-to-release-2.1
• cherry-pick-2904-to-release-2.1
• cherry-pick-2907-to-release-2.1
• cherry-pick-2908-to-release-2.1
• cherry-pick-2913-to-release-2.1
• cherry-pick-2923-to-release-2.1
• cherry-pick-2926-to-release-2.1
• cherry-pick-2971-to-release-2.1
• cherry-pick-3009-to-release-2.1
• cherry-pick-3010-to-release-2.1
• cherry-pick-changelog-1.9
• chore/KEP-runtime-class
• chore/gha
• chore/merge-podspacoverride-test-cases
• chore/upgrade-torchtune-version
• ci/include-1.32-k8s
• claude-symlink
• code-quality-check
• code-quality-clean
• config-api-implementation
• coscheduling-indexers-ut
• deepspeed-runtime
• dependabot/cargo/pkg/data_cache/arrow-56.2.0
• dependabot/cargo/pkg/data_cache/arrow-57.0.0
• dependabot/cargo/pkg/data_cache/arrow-57.1.0
• dependabot/cargo/pkg/data_cache/arrow-57.2.0
• dependabot/cargo/pkg/data_cache/arrow-flight-56.2.0
• dependabot/cargo/pkg/data_cache/arrow-flight-57.1.0
• dependabot/cargo/pkg/data_cache/arrow-flight-57.2.0
• dependabot/cargo/pkg/data_cache/arrow-schema-56.2.0
• dependabot/cargo/pkg/data_cache/arrow-schema-57.2.0
• dependabot/cargo/pkg/data_cache/async-trait-0.1.89
• dependabot/cargo/pkg/data_cache/axum-0.8.8
• dependabot/cargo/pkg/data_cache/bincode-2.0.1
• dependabot/cargo/pkg/data_cache/bincode-3.0.0
• dependabot/cargo/pkg/data_cache/bytes-1.11.0
• dependabot/cargo/pkg/data_cache/bytes-1.11.1
• dependabot/cargo/pkg/data_cache/crossbeam-channel-0.5.15
• dependabot/cargo/pkg/data_cache/datafusion-51.0.0
• dependabot/cargo/pkg/data_cache/futures-0.3.32
• dependabot/cargo/pkg/data_cache/hickory-resolver-0.25.2
• dependabot/cargo/pkg/data_cache/iceberg-0.6.0
• dependabot/cargo/pkg/data_cache/iceberg-0.7.0
• dependabot/cargo/pkg/data_cache/iceberg-0.8.0
• dependabot/cargo/pkg/data_cache/iceberg-datafusion-0.6.0
• dependabot/cargo/pkg/data_cache/iceberg-datafusion-0.7.0
• dependabot/cargo/pkg/data_cache/iceberg-datafusion-0.8.0
• dependabot/cargo/pkg/data_cache/ring-0.17.14
• dependabot/cargo/pkg/data_cache/serde-1.0.228
• dependabot/cargo/pkg/data_cache/test/arrow-flight-57.0.0
• dependabot/cargo/pkg/data_cache/test/arrow-flight-57.1.0
• dependabot/cargo/pkg/data_cache/test/arrow-flight-57.2.0
• dependabot/cargo/pkg/data_cache/test/arrow-flight-57.3.0
• dependabot/cargo/pkg/data_cache/test/bincode-2.0.1
• dependabot/cargo/pkg/data_cache/test/bincode-3.0.0
• dependabot/cargo/pkg/data_cache/test/bytes-1.11.0
• dependabot/cargo/pkg/data_cache/test/bytes-1.11.1
• dependabot/cargo/pkg/data_cache/test/clap-4.5.51
• dependabot/cargo/pkg/data_cache/test/clap-4.5.52
• dependabot/cargo/pkg/data_cache/test/clap-4.5.53
• dependabot/cargo/pkg/data_cache/test/clap-4.5.54
• dependabot/cargo/pkg/data_cache/test/clap-4.5.56
• dependabot/cargo/pkg/data_cache/test/clap-4.5.57
• dependabot/cargo/pkg/data_cache/test/clap-4.5.59
• dependabot/cargo/pkg/data_cache/test/futures-0.3.32
• dependabot/cargo/pkg/data_cache/test/serde-1.0.228
• dependabot/cargo/pkg/data_cache/test/tokio-1.48.0
• dependabot/cargo/pkg/data_cache/test/tokio-1.49.0
• dependabot/cargo/pkg/data_cache/test/tonic-0.14.2
• dependabot/cargo/pkg/data_cache/test/tonic-0.14.3
• dependabot/cargo/pkg/data_cache/test/tonic-0.14.4
• dependabot/cargo/pkg/data_cache/test/tracing-0.1.43
• dependabot/cargo/pkg/data_cache/test/tracing-0.1.44
• dependabot/cargo/pkg/data_cache/test/tracing-subscriber-0.3.20
• dependabot/cargo/pkg/data_cache/test/tracing-subscriber-0.3.22
• dependabot/cargo/pkg/data_cache/time-0.3.47
• dependabot/cargo/pkg/data_cache/tokio-1.44.2
• dependabot/cargo/pkg/data_cache/tokio-1.48.0
• dependabot/cargo/pkg/data_cache/tokio-1.49.0
• dependabot/cargo/pkg/data_cache/tonic-0.14.2
• dependabot/cargo/pkg/data_cache/tonic-0.14.3
• dependabot/cargo/pkg/data_cache/tonic-0.14.4
• dependabot/cargo/pkg/data_cache/tower-0.5.2
• dependabot/cargo/pkg/data_cache/tower-0.5.3
• dependabot/docker/cmd/data_cache/rust-1.91-bullseye
• dependabot/docker/cmd/data_cache/rust-1.92-bullseye
• dependabot/docker/cmd/data_cache/rust-1.93-bullseye
• dependabot/docker/cmd/initializers/dataset/python-3.14-slim-bookworm
• dependabot/docker/cmd/initializers/model/python-3.14-slim-bookworm
• dependabot/docker/cmd/runtimes/deepspeed/mpioperator/base-v0.7.0
• dependabot/docker/cmd/runtimes/deepspeed/nvidia/cuda-13.0.2-devel-ubuntu22.04
• dependabot/docker/cmd/runtimes/deepspeed/nvidia/cuda-13.1.0-devel-ubuntu22.04
• dependabot/docker/cmd/runtimes/deepspeed/nvidia/cuda-13.1.1-devel-ubuntu22.04
• dependabot/docker/cmd/runtimes/mlx/mpioperator/base-v0.7.0
• dependabot/docker/cmd/runtimes/mlx/nvidia/cuda-13.0.2-devel-ubuntu22.04
• dependabot/docker/cmd/runtimes/mlx/nvidia/cuda-13.1.0-devel-ubuntu22.04
• dependabot/docker/cmd/runtimes/mlx/nvidia/cuda-13.1.1-devel-ubuntu22.04
• dependabot/docker/cmd/trainer-controller-manager/golang-1.25
• dependabot/docker/cmd/trainer-controller-manager/golang-1.26
• dependabot/docker/cmd/trainers/torchtune/pytorch/pytorch-2.10.0-cuda12.8-cudnn9-runtime
• dependabot/docker/cmd/trainers/torchtune/pytorch/pytorch-2.9.0-cuda12.8-cudnn9-runtime
• dependabot/docker/cmd/trainers/torchtune/pytorch/pytorch-2.9.1-cuda12.8-cudnn9-runtime
• dependabot/github_actions/actions/checkout-5
• dependabot/github_actions/actions/checkout-6
• dependabot/github_actions/actions/github-script-8
• dependabot/github_actions/actions/setup-go-6
• dependabot/github_actions/actions/setup-python-6
• dependabot/github_actions/actions/stale-10
• dependabot/github_actions/actions/upload-artifact-5
• dependabot/github_actions/actions/upload-artifact-6
• dependabot/github_actions/amannn/action-semantic-pull-request-6.1.1
• dependabot/github_actions/aquasecurity/trivy-action-0.33.1
• dependabot/github_actions/aquasecurity/trivy-action-0.34.0
• dependabot/github_actions/dot-github/workflows/aquasecurity/trivy-action-0.34.0
• dependabot/github_actions/github/codeql-action-4
• dependabot/go_modules/github.com/onsi/ginkgo/v2-2.27.2
• dependabot/go_modules/github.com/onsi/ginkgo/v2-2.27.3
• dependabot/go_modules/github.com/onsi/ginkgo/v2-2.27.5
• dependabot/go_modules/github.com/onsi/ginkgo/v2-2.28.1
• dependabot/go_modules/github.com/onsi/gomega-1.38.3
• dependabot/go_modules/github.com/onsi/gomega-1.39.0
• dependabot/go_modules/github.com/onsi/gomega-1.39.1
• dependabot/go_modules/github.com/open-policy-agent/cert-controller-0.15.0
• dependabot/go_modules/go.uber.org/zap-1.27.1
• dependabot/go_modules/golang-c94709d3c3
• dependabot/go_modules/golang-ce64870c5e
• dependabot/go_modules/golang-cf2caa1bb8
• dependabot/go_modules/golang-edfadaf7f0
• dependabot/go_modules/golang-f180a085e8
• dependabot/go_modules/golang.org/x/crypto-0.45.0
• dependabot/go_modules/golang.org/x/net-0.38.0
• dependabot/go_modules/golang.org/x/oauth2-0.27.0
• dependabot/go_modules/kubernetes-2b83cfd1e1
• dependabot/go_modules/kubernetes-33780c5637
• dependabot/go_modules/kubernetes-46bc08174d
• dependabot/go_modules/kubernetes-bd430bb9c9
• dependabot/go_modules/kubernetes-df4453129a
• dependabot/go_modules/kubernetes-e0300699ac
• dependabot/go_modules/kubernetes-faa114bc83
• dependabot/go_modules/kubernetes-fdea40109e
• dependabot/pip/cmd/initializers/dataset/huggingface-hub-gte-0.27.0-and-lt-1.2
• dependabot/pip/cmd/initializers/dataset/huggingface-hub-gte-0.27.0-and-lt-1.3
• dependabot/pip/cmd/initializers/dataset/huggingface-hub-gte-0.27.0-and-lt-1.4
• dependabot/pip/cmd/initializers/dataset/huggingface-hub-gte-0.27.0-and-lt-1.5
• dependabot/pip/cmd/initializers/model/huggingface-hub-gte-0.27.0-and-lt-1.2
• dependabot/pip/cmd/initializers/model/huggingface-hub-gte-0.27.0-and-lt-1.3
• dependabot/pip/cmd/initializers/model/huggingface-hub-gte-0.27.0-and-lt-1.4
• dependabot/pip/cmd/initializers/model/huggingface-hub-gte-0.27.0-and-lt-1.5
• dependabot/pip/cmd/runtimes/deepspeed/datasets-4.4.1
• dependabot/pip/cmd/runtimes/deepspeed/datasets-4.4.2
• dependabot/pip/cmd/runtimes/deepspeed/datasets-4.5.0
• dependabot/pip/cmd/runtimes/deepspeed/deepspeed-0.18.2
• dependabot/pip/cmd/runtimes/deepspeed/deepspeed-0.18.3
• dependabot/pip/cmd/runtimes/deepspeed/deepspeed-0.18.4
• dependabot/pip/cmd/runtimes/deepspeed/deepspeed-0.18.5
• dependabot/pip/cmd/runtimes/deepspeed/deepspeed-0.18.6
• dependabot/pip/cmd/runtimes/deepspeed/mpi4py-4.1.1
• dependabot/pip/cmd/runtimes/deepspeed/sentencepiece-0.2.1
• dependabot/pip/cmd/runtimes/deepspeed/torch-2.10.0
• dependabot/pip/cmd/runtimes/deepspeed/torch-2.6.0
• dependabot/pip/cmd/runtimes/deepspeed/torch-2.7.1
• dependabot/pip/cmd/runtimes/deepspeed/torch-2.8.0
• dependabot/pip/cmd/runtimes/deepspeed/torch-2.9.0
• dependabot/pip/cmd/runtimes/deepspeed/torch-2.9.1
• dependabot/pip/cmd/runtimes/deepspeed/transformers-4.51.0
• dependabot/pip/cmd/runtimes/deepspeed/transformers-4.52.1
• dependabot/pip/cmd/runtimes/deepspeed/transformers-4.53.0
• dependabot/pip/cmd/runtimes/deepspeed/transformers-4.57.1
• dependabot/pip/cmd/runtimes/deepspeed/transformers-4.57.2
• dependabot/pip/cmd/runtimes/deepspeed/transformers-4.57.3
• dependabot/pip/cmd/runtimes/deepspeed/transformers-4.57.6
• dependabot/pip/cmd/runtimes/deepspeed/transformers-5.0.0
• dependabot/pip/cmd/runtimes/deepspeed/transformers-5.1.0
• dependabot/pip/cmd/runtimes/deepspeed/transformers-5.2.0
• dependabot/pip/cmd/runtimes/mlx/datasets-4.4.1
• dependabot/pip/cmd/runtimes/mlx/datasets-4.4.2
• dependabot/pip/cmd/runtimes/mlx/datasets-4.5.0
• dependabot/pip/cmd/runtimes/mlx/mlx-cuda--0.29.3
• dependabot/pip/cmd/runtimes/mlx/mlx-cuda--0.30.0
• dependabot/pip/cmd/runtimes/mlx/mlx-cuda--0.30.1
• dependabot/pip/cmd/runtimes/mlx/mlx-cuda--0.30.3
• dependabot/pip/cmd/runtimes/mlx/mlx-cuda--0.30.5
• dependabot/pip/cmd/runtimes/mlx/mlx-cuda--0.30.6
• dependabot/pip/cmd/runtimes/mlx/mlx-data-0.2.0
• dependabot/pip/cmd/runtimes/mlx/mlx-lm-0.28.3
• dependabot/pip/cmd/runtimes/mlx/mlx-lm-0.28.4
• dependabot/pip/cmd/runtimes/mlx/mlx-lm-0.30.0
• dependabot/pip/cmd/runtimes/mlx/mlx-lm-0.30.2
• dependabot/pip/cmd/runtimes/mlx/mlx-lm-0.30.4
• dependabot/pip/cmd/runtimes/mlx/mlx-lm-0.30.5
• dependabot/pip/cmd/runtimes/mlx/mlx-lm-0.30.6
• dependabot/pip/cmd/trainers/torchtune/torchao-0.16.0
• docs/local-examples-gpu-support
• docs/local-iceberg-validation
• dont-merge-gpu-label-test
• example/trainjob-yaml
• feat/add-coscheduling-uts
• feat/add-helm-ci-checks
• feat/add-securitycontext-support-trainjob
• feat/add-version-file
• feat/automate-release
• feat/config-unit-tests
• feat/ctr-webhook
• feat/dataset-preprocess
• feat/example/add-speech-recognition-with-ddp-example
• feat/helm-data-cache-config
• feat/initializers/s3
• feat/llama3_2-manifests
• feat/llm-trainer-v2
• feat/local-model
• feat/lora-support
• feat/pvc-check
• feat/replica-valid
• feat/sdk-torchtune-config
• feat/torchtune-plugin
• feat/trainjob-affinity
• feat/trainjob-imagepullsecrets
• feat/webhook-validate-trainjob-name
• feat/webhook/rfc1035
• feature-gate-scaffolding
• feature/add-xgboost-runtime
• feature/debabrata
• feature/helm-charts-v2
• feature/kube-linter-3096
• feature/support-for-ClusterTrainingRuntimes
• fix-arg-for-get-args-using-torchtune-config
• fix-close-pr-message
• fix-controller-rbac
• fix-coveralls
• fix-crd-cel-namespace
• fix-deepspeed-example
• fix-deepspeed-npoc
• fix-e2e-sdk-install
• fix-example-runtime
• fix-helm-chart-name
• fix-helm-charts-config-api-2894
• fix-helm-lint
• fix-helm-unittest-logic-clean
• fix-immutable-apis
• fix-kep-volcano
• fix-latest-dev
• fix-latest-tag
• fix-llm-hp-optimization-error
• fix-local-tests
• fix-mlx-runtime
• fix-mpi-key-mode
• fix-oci-vm-tf
• fix-outdated-intstr-lib
• fix-permissions
• fix-readonly-rootfs
• fix-release-doc
• fix-resource-allocation
• fix-serviceaccount-name
• fix-suspend-resume-3008
• fix-tag-manager
• fix-test-bug
• fix-torchtune-runtime-deps
• fix-trainer-type-annotation
• fix/allow-podtemplate-update-on-unsuspend
• fix/cert-and-issuer
• fix/ci-duplicate-step-name
• fix/disable-github-actions
• fix/helm-chart
• fix/issue-template
• fix/jax-validation
• fix/kep2401-lint
• fix/multiple-depends-on
• fix/python-type-import
• fix/rbac/event
• fix/remove-jobset-lws-patches
• fix/runtime-info-thread-safety
• fix/sync-podsets-count-to-template-spec
• fix/tidy-KEP-2401
• fix/torchtune-c-compiler
• fix/torchtune-plugin
• flux-framework-plugin
• gpu-arc-doc
• gpu-test-on-pr
• gsoc-2442-jax-runtime-proposal
• gsoc-pss-istio-fix
• gsoc25-project7-kep
• hatchling-package
• health
• helm-integration-tests
• implement-resource-in-use-finalizer
• implement-resource-in-use-for-cl-training-runtime
• implement-validation-uts
• indexers-ut
• issue-2218-pod-spec-override-kep
• issue-2547
• issue-2706-v2-go-mod
• issue-2789/implement-cluster-training-runtimes-deprecation-process
• jax-runtime
• jax-runtime-impl
• jobset-name-prefix
• jobset-validation
• jobset-volume-claim-policies
• k8s_1.32_upgrade
• kai-scheduler-2628
• kai_kep
• kep-2598-xgboost-runtime
• kep-2779-trainjob-progress
• kep-2841-add-flux-hpc
• kubecon-london-demo
• kubelow-sdk-release
• master
• megatron
• mlx-cuda-runtime
• mlx-runtime
• move-imports
• obtain-runtimeTemplate-via-info
• openssf-badge
• override_label_and_annotation
• patch-1
• patch-issue-2027
• pick/example-alpaca
• pick/fix-torchtune-plugin
• pkg/apply_unit-tests
• plugin/flux
• pr-15
• pr-17
• pr-18
• pr-19
• pr-20
• pr-21
• pr-22
• pr-24
• pr-25
• pr-26
• pr-27
• pr-28
• pr-29
• pr-30
• pr-32
• pr-33
• pr-35
• pr-36
• pr-37
• pr-38
• pr-39
• pr-41
• pr-42
• pr-43
• pr-44
• pr-45
• pr-created-condition
• pr-k8s-lint
• pr-title-workflow
• prometheus
• proposal
• proposal-2170
• pss-restricted-fixes
• public-configmap
• refactor-named-container-ports
• refs/tags/v1.9.1
• refs/tags/v2.0.0-rc.0
• refs/tags/v2.0.0-rc.1
• refs/tags/v2.0.1
• refs/tags/v2.1.0
• refs/tags/v2.1.0-rc.0
• refs/tags/v2.1.0-rc.1
• release-1.9
• release-2.0
• release-2.1
• release-python-doc
• remove-command-runtimes
• remove-k8s-version-matrix
• remove-mpi
• remove-sdk
• remove-vendor-specific-parameters
• revert-2646-fix-trainer-type-annotation
• roadmap-2025
• rqst-env-only-if-label-present
• runtime-rbac
• runtime_fix
• safe-gpu-test
• scorecard-workflow
• sdk-ancestor-updates
• sdk-fix-mpirun
• security-doc
• separate-models-from-sdk
• solanyn/question-answer-example
• support-arm-container
• support-for-gpu-cluster-using-oci-runner
• support_kai
• terrytangyuan-patch-1
• test-cncf-gpu-runner
• test-gpu-arc
• test/fix-flaky-test
• test/runtime-core-coverage
• tmp_secret_verify
• training-progression#2779
• treat-ancestor-label-as-identifier
• trivy-scans
• update-approvers
• update-examples-with-unpacking-params
• update-github-runners
• update-image-tags
• update-jobset-0.11
• update-license
• update-logs-examples
• update-manifest-images-to-ghcr
• update-news-v2.2-release
• update-owners
• update-readme
• update-release-process
• update-sdk-reference
• update-security-context
• update-slack
• update-stale-bot-version
• update-torch-2.9
• use-tilt
• validation-mpiruntimes
• volcano
• volcano-podgroup-build
• vuls
• vzamboulingame-upgrade-go-v1.24
• was-kep
• workflow/helm
• workflow/publish-helm-charts
• xgboost-runtime-implementation

Build # 22192323106

Build Type

Pull #3227

github

Committed by robert-bell

Commit Message

chore: fix config tests after rebase

Signed-off-by: Rob Bell <robell@redhat.com>

Pull Request Pull Request #3227: [WIP] feat: add support for tracking TrainJob progress and training metrics

Run Details

284 of 503 new or added lines in 11 files covered. (56.46%)

1675 of 2977 relevant lines covered (56.26%)

0.66 hits per line