kubeflow / trainer / 39539188200

77%

Build Type

push

github

Committed by web-flow

Commit Message

feat(framework): clean up Volcano PodGroup on TrainJob terminal or suspended state (#4059)

* feat(framework): clean up Volcano PodGroup on TrainJob terminal or suspended state

Add a new optional ComponentDeleterPlugin interface, symmetric to the
existing ComponentBuilderPlugin: a plugin returns objects that should
now be removed for a TrainJob, and the reconciler performs the actual
deletion. It runs on every reconcile regardless of the TrainJob's
status, so a plugin decides for itself whether an object should
currently exist, and must be idempotent.

Volcano implements it to delete its PodGroup once the TrainJob is
Failed, Complete, or suspended, so it stops reserving minResources
capacity in the Volcano queue indefinitely. Volcano.Build() is updated
to skip (re)creating the PodGroup while suspended, so its lifecycle
during suspension is fully owned by Delete() rather than racing with
it.

Adds delete on scheduling.volcano.sh/podgroups to the controller RBAC,
which was missing.

Fixes #4033

Signed-off-by: daehyun-kim-94 <daehyun.kim.0613@gmail.com>

* fix(framework): address review feedback on Volcano PodGroup cleanup

Addresses andreyvelich's review on #4059:

- reconcileObjects() always runs DeleteObjects even when NewObjects or
  Apply fails, joining both errors so delete failures are distinguishable
  under TrainJobResourcesCreationFailed.
- TrainingRuntime/ClusterTrainingRuntime.DeleteObjects log and return nil
  instead of erroring when the runtime can no longer be resolved or its
  Info can't be built, since that path also runs for finished TrainJobs.
  getTrainingRuntime/getClusterTrainingRuntime share the snapshot/live
  lookup with NewObjects without creating a snapshot on the delete path.
- Volcano.Build()'s suspend-skip comment is corrected: inputs can change
  while suspended, so skipping updates there is safe because Delete()
  removes the PodGroup and Build() rebuilds it fresh on resume.
- Removed the now-unused PodGroupRuntimeClass... (continued)

Coverage Stats

55 of 61 new or added lines in 4 files covered. (90.16%)

1 existing line in 1 file now uncovered.

3234 of 4202 relevant lines covered (76.96%)

0.77 hits per line