kubeflow / training-operator

40%

master: 100%

LAST BUILD ON BRANCH mpijob-fix

branch: mpijob-fix

CHANGE BRANCH

x

Reset

Sync Branches

• mpijob-fix
• 1.6release
• 1570
• 2005
• Training-operator-ppc64le
• add-apigrop-to-training-runtime-ref
• add-cherry-pick-script
• add-clientset
• add-coscheduling-e2e
• add-e2e-test-for-train-api
• add-example-fine-tune-llm
• add-flake8-check
• add-image-build-ci
• add-meeting-readme
• add-mpi-demo
• add-overlay-manifest-v2
• add-pr-template
• add-stale-action
• add-trainjob-state-transition
• add-validation-for-service
• add_k8s_env_vars
• apireader
• automated-cherry-pick-of-#2079-#2130-upstream-v1.8-branch
• automated-cherry-pick-of-#2079-upstream-v1.8-branch
• automated-cherry-pick-of-#2105-#2109-#2122-#2079-#2130-upstream-v1.8-branch
• automated-cherry-pick-of-#2105-#2109-#2122-upstream-v1.8-branch
• automated-cherry-pick-of-#2105-#2122-#2079-#2130-upstream-v1.8-branch
• automated-cherry-pick-of-#2105-#2122-upstream-v1.8-branch
• automated-cherry-pick-of-#2106-upstream-v1.8-branch
• automated-cherry-pick-of-#2109-upstream-v1.8-branch
• avoid-copy-repository
• bug/broken-preemption-on-suspended-with-ttl
• bump-crd-ref-gen-version
• bump-go-version
• bump-sdk-1.8.0
• bump-sdk-1.8.1
• cel-crd
• change-grab-code-gen
• change-supported-python-version
• change-v2-manifests
• changelog-1.8.0
• changelog-1.8.0-rc.0
• changelog-1.8.0-rc.1
• changelog-1.8.1
• changelog-1.9.0
• changelog-1.9.0-rc0
• check-crd
• cherry-pick-2369-to-release-1.9
• cherry-pick-2376-to-release-1.9
• cherry-pick-2385-to-release-1.9
• cherry-pick-2391-to-release-1.9
• cherry-pick-2399-to-release-1.9
• cherry-pick-from-1733
• cherry-pick-of-2243
• clean-up-commonutils
• clean-up-go-module
• clean-up-status-util
• com
• commonize-pointer-cleanPodPolicy
• commonize-timeout
• commonize-webhook-validations
• conditions-unit-tests
• config
• consistent-naming
• containerize_e2e_tests
• context-error
• coscheduling
• cr-finalizers
• darwin-support
• debug
• debug_e2e
• decouple-jobset-and-trainjob
• deepspeed
• default_replicas
• delete-deadcode-pdb
• dep
• dependabot
• dependabot/go_modules/golang.org/x/net-0.17.0
• dependabot/go_modules/golang.org/x/net-0.33.0
• dependabot/go_modules/golang.org/x/net-0.7.0
• dependabot/go_modules/google.golang.org/protobuf-1.33.0
• dependabot/go_modules/hack/swagger/github.com/emicklei/go-restful/v3-3.8.0
• dependabot/go_modules/hack/swagger/golang.org/x/net-0.17.0
• dependabot/go_modules/hack/swagger/golang.org/x/net-0.7.0
• dependabot/go_modules/hack/swagger/golang.org/x/text-0.3.8
• dependabot/go_modules/hack/swagger/google.golang.org/protobuf-1.33.0
• dependabot/pip/examples/pytorch/deepspeed-demo/tqdm-4.66.3
• dependabot/pip/examples/pytorch/deepspeed-demo/transformers-4.38.0
• dependabot/pip/examples/xgboost/lightgbm-dist/numpy-1.22.0
• dependabot/pip/examples/xgboost/lightgbm-dist/scikit-learn-1.0.1
• dependabot/pip/examples/xgboost/lightgbm-dist/scikit-learn-1.5.0
• dependabot/pip/examples/xgboost/lightgbm-dist/scipy-1.10.0
• dependabot/pip/py/kubeflow/tf_operator/certifi-2022.12.7
• dependabot/pip/py/kubeflow/tf_operator/httplib2-0.19.0
• dependabot/pip/py/kubeflow/tf_operator/jinja2-2.11.3
• dependabot/pip/py/kubeflow/tf_operator/oauthlib-3.2.1
• dependabot/pip/py/kubeflow/tf_operator/protobuf-3.15.0
• dependabot/pip/py/kubeflow/tf_operator/protobuf-3.18.3
• dependabot/pip/py/kubeflow/tf_operator/pyyaml-5.4
• dependabot/pip/py/kubeflow/tf_operator/rsa-4.7
• dependabot/pip/sdk/python/black-24.3.0
• dependabot/pip/sdk/python/kubeflow/storage_initializer/transformers-4.38.0
• dependabot/pip/sdk/python/kubeflow/trainer/transformers-4.38.0
• deprecated
• dir-typo
• distributed-data-train
• dnplas-jaxjobs-rbac-2416
• doc/KEP-2401
• docs
• docs/dev-guide-update
• docs/remove-monitoring-doc
• dynamic-worker-fix
• e2e-test-in-cluster
• enh/bump-k8s-deps
• example-pytorch-cpu
• fangyuxuan/fix_mpi_bug
• feat/example-xgboost-arm64
• feature/helm-charts
• feature/requeue_on_create_status
• feature/success_failure_policy
• feature/support-managed-by-external-controller
• feature/support_monitor_podgroup
• filter-by-namespace
• finalizers
• first-implementaton-for-traininig-v2
• fitComments
• fix
• fix-apiClient
• fix-ci
• fix-ci-push-branch
• fix-deepspeed
• fix-doc-img
• fix-e2e-tests
• fix-example-indent
• fix-flaky-ttl-test
• fix-gang-scheduling-e2e
• fix-get-pvc-spec
• fix-github-action-script
• fix-is-master
• fix-issue-pr
• fix-jobstatus-update
• fix-missspell
• fix-mpi
• fix-mpi-controller
• fix-mpijob-watch-source
• fix-nproc
• fix-options
• fix-pr
• fix-predicators
• fix-pytorch-ddp
• fix-sdk-events
• fix-sdk/getch-base-image
• fix-status-lost
• fix-tf-operator-link
• fix-torch-env
• fix-train-example-import
• fix-trainer-error
• fix-typo
• fix-v2-startup-error
• fix-version-transformer
• fix-volcano-panic
• fix-xgboost-bug
• fix-xgboost-flaky-tests
• fix-xgboostjob-controller
• fix/add-start-time
• fix/build-mnist-img-err
• fix/ci_integration_tests
• fix/clean-up-unused-mpi-code
• fix/mpi-unable-to-use-gang-scheduling
• fix/printcolumn
• fix/set_scheduler_error
• fix/tfjob-add-warn-for-failed-validation
• fix/tfjob-restartpolicy-exitcode-not-work
• fix/update-mpi-unit-test
• fix/webhook-server-port-flag
• fixNullQueue
• fix_ci_workflow
• fix_cleanpodpolicy
• fix_doc
• fix_elastic
• fix_loop
• fix_rbac
• fix_xgb_example_yaml
• fixzapr
• fsdp-example-torch
• fully-consolidate-tfjob
• gaocegege-patch-1
• geenrate-crds-v2
• generate/v2-clients
• gha
• gloo-build-failure
• helenxie/add-parameters-to-helper-function
• helenxie/fix-trainer-error
• helenxie/fix_huggingface_hub_version
• helenxie/update-name-of-pvc
• helenxie/update-trainer
• hpa
• ignore-cache-exporting-error
• image-latest
• imagebuild
• imgup
• implemenet-webhook-validations
• implement-mxjob-webhook-warnings
• implement-paddle-webhook-validations
• implement-sckelton-webhook-servers
• implement-supend-test-for-mxjob
• implement-supend-test-for-xgboostjob
• implement-suspend-mpijob
• implement-suspend-test-for-tfjob
• implement-suspend-test-to-paddlejob
• implement-tfjob-webhook-validations
• implement-trainjob-reconciler
• implement-xgboost-webhook-validation
• improve
• improve-e2e-test
• improve-readme
• improve-startup-time
• increase-ci-root-disk
• increase-success-threshold
• initializer-test
• intel-mpi-env-variables
• isort-fix
• issue-1436-add-go-test-to-ci
• issue-1863-sdk-job-events
• issue-1877-consolidate-sdk-apis
• issue-1946-use-logging-sdk
• issue-1991-sdk-torchrun
• issue-2206-add-apis
• issue-2206-api-initialization
• issue-2208-v2-manifests
• issue-2210-storage-initializer
• issue-2211-add-torch-runtime
• issue-2216-initial-sdk-api
• issue-2216-v2-sdk-generation
• issue-2217-update-kep-mpi
• issue-2290-init-builder
• issue-2290-torch-plugin
• issue-2367-fix-v1-storage-init
• issue-template-triage
• issue_1924
• issues1704
• issues1706
• jax-controller
• jax-example
• jax-integration-proposal
• jaxapi
• job-status-optional
• koordinator
• kubectl-delivery
• kustomize-v5
• latest-image
• launcher-service-account
• linguist-fix
• log-level
• make-xgb-image-community-driven
• makefile
• master
• master-pr
• mod-upgrade-to-ginkgo-v2
• mpi-api-doc
• mxnet
• owners
• paddle-elastic
• paddlepaddle
• pass_timeout_from_tests
• patch-1
• patch-2
• patch-issue-2016
• patch-issue-2016-remove-dockerfile-ppc64le
• patch-issue-2025
• patch-issue-2048
• patch/issue-2050
• pc
• pc-lint
• pin-pkg
• pkg-import
• pod_policy
• pr-01
• pr-02
• pr-03
• pr-04
• pr-05
• pr-06
• pr-07
• pr-08
• pr-09
• pr-10
• pr-12
• pr-k8s-1.30
• pr-k8s-1.31
• pr_ttl
• profile_switch
• push-mxnet-image-to-registry
• py-sdk-fix-job-role-label
• pybuffer
• python-lint-format
• python-sdk-path
• pytorch-is-PyTorch
• pytorch-job-notebook-test
• pytorch_ismaster_fix
• pytorchelastic
• reduce/exception
• refactor-core-pod-test
• refactor-swagger-generation
• refactor/v2alpha-api
• refine-installation-docs
• refine-pytorchjob-integration-test
• refs/tags/v1.4.0-rc.0
• refs/tags/v1.5.0
• refs/tags/v1.5.0-rc.0
• refs/tags/v1.6.0
• refs/tags/v1.6.0-rc.0
• refs/tags/v1.6.0-rc.1
• refs/tags/v1.7.0
• refs/tags/v1.7.0-rc.0
• refs/tags/v1.8.0
• refs/tags/v1.8.0-rc.0
• refs/tags/v1.8.0-rc.1
• refs/tags/v1.8.1
• refs/tags/v1.9.0-rc.0
• release-1.9
• remove-apache-mxnet
• remove-klog-v1
• remove-testJobReconciler
• remove-v1-code
• replace-commonv1-with-kubeflowv1
• replace-dummy-client-with-fake-client
• replace-outdated-pytorch-images-in-ci
• rm-mxnet
• run-various-python-versions
• runtimecel
• sdk-doc-train-api
• sdk-fix-create-replicas
• sdk-fix-failed-condition
• sdk-no-config-flag
• sdk-python-version
• sdk-read-namespace-context
• sdk-remove-final-keyword
• sdk-resource-per-worker
• sdk-test-create-from-func
• sdk-unify-training-client
• sdk/fetch-base-image
• sdkup
• second-implementation-for-traininig-v2
• service-account-fix
• show-warning-pytorchjob-nproc
• simplify-conditions-compare
• soc
• sparse_cluster
• stop-cache-max
• style/replica-name
• support-arm64-for-huggingface-trainer
• support-configure-qps-and-burst
• support-k8s-128
• support-k8s-129
• support-single
• support-suspend-semantics-for-pytorchjob
• suspend-semantic
• tensorflow-example
• tensorflow-v2-upgrade
• terrytangyuan-patch-1
• terrytangyuan-patch-2
• test-new-common
• test-pr
• test/wait-for-job-conditions
• testutil
• torch-env-fix
• train_api
• train_api_dataset
• train_api_example_fix
• train_api_hf_dataset
• train_api_image_name_fix
• train_api_models
• train_api_proposal
• train_api_storage
• train_api_training_image
• train_api_utils_change
• training-role-mpijob
• training-v2-proposal
• trigger-gha-workflow
• trim-down-crd-description
• unit-test-delete-job
• unit-test-get-job
• unit_test-TrainingClient-get_job_logs
• unit_test-TrainingClient-get_job_pods
• unit_test-TrainingClient-list_jobs
• unit_test-TrainingClient-new_tests
• update-api-v2-kep-2206
• update-changelog
• update-dev-docs
• update-dev-guide
• update-diagram
• update-examples
• update-jax-image
• update-jax-proposal
• update-kep-2170
• update-label
• update-owners
• update-pytorch-sdk-create-from-func
• update-readme
• update-release-doc
• update-slack-link
• update-xgboost--e2e-image
• update_TFJob_examples_to_TF_v2
• upgrade-actions-checkout-version
• upgrade-go-version
• upgrade-k8s-1.27
• upgrade-k8s-v1.26
• upgrade-kubernetes-version
• upgrade-pytorch-version
• upgrade-scheduler-plugins
• upgrade-scheduler-plugins-version
• upgrade-sdk-k8s-minimul-requirements
• upgrade-setup-env-version
• use-community-hosted-mxnet-image
• use-eventually-tf-status-test
• use-same-reason
• ut-mxnet
• ut-tf
• ut-xgboost
• v1.4-branch
• v1.5-branch
• v1.6-branch
• v1.7
• v1.7-branch
• v1.8-branch
• v2-add-reconcile-condition
• v2-refactor-webhook-integration-test
• v2-rename-runtime-ref
• validate-workers-for-elasticpolicy
• volcano-e2e
• volcano-release
• volcano-support
• wait_for_job_conds_bug_fix
• webhookv2
• workshop
• wyen/fix_pg_update
• xgboost-examples

fix MpiJob status inaccuracy With elastic training enabled, when some worker Pods is failed, evicted or deleted, the MPIJob should continue to run.