kubeflow / trainer / 25358512691

58%

push

github

fix(api): allow atomic RuntimePatches update on unsuspend (#3469)

* fix(webhook): allow atomic RuntimePatches update on unsuspend

Change checkRuntimePatchesImmutability to check the *old* TrainJob's
suspend state instead of the new one. This lets external controllers
(notably Kueue, see kubernetes-sigs/kueue#8296) modify spec.runtimePatches
and set spec.suspend=false in a single API request, eliminating the
two-step update workaround and the race window between the calls.

Mirrors the Kubernetes core Job validation pattern:
https://github.com/kubernetes/kubernetes/blob/86b66f6f333a/pkg/registry/batch/job/strategy.go#L191

The downstream JobSet activity check (no active replicatedJobs) still
runs, so updates while pods are running remain forbidden.

Refs: #3043
Signed-off-by: Sabari Narayana <sabarinarayanakg@proton.me>

* test(jobset): cover atomic suspend->unsuspend RuntimePatches update

Add a unit test that exercises the new atomic-update path: a single
TrainJob update that both clears spec.suspend and modifies
spec.runtimePatches must succeed when the previous state was suspended.

Refs: #3043
Signed-off-by: Sabari Narayana <sabarinarayanakg@proton.me>

* test(integration): cover atomic unsuspend + RuntimePatches update

Extend the trainjob webhook update table with an entry that creates a
suspended TrainJob with RuntimePatches and then, in a single Update,
both unsuspends it and modifies the runtimePatches NodeSelector. This
exercises the relaxed validation against a real apiserver and protects
the suspended -> unsuspended transition path used by Kueue.

Refs: #3043
Signed-off-by: Sabari Narayana <sabarinarayanakg@proton.me>

* fix(api): block RuntimePatches change only when both states are unsuspended

Per @mimowo's review, the previous condition (!oldSuspended) over-blocked
the (oldSuspended=false, newSuspended=true) transition — i.e. suspending
a running TrainJob while modifying spec.runtimePatches in the same call,
which is safe because the... (continued)

8 of 8 new or added lines in 1 file covered. (100.0%)

2037 of 3505 relevant lines covered (58.12%)

0.67 hits per line