kubeflow / trainer / 31531954434

68%

push

github

fix(api): reject zero and negative numNodes and numProcPerNode (#3809)

TrainJob.Spec.Trainer.NumNodes and NumProcPerNode had no lower-bound
validation, so values like numNodes: -3 or numProcPerNode: -5 were
accepted at admission and propagated into the pod spec (the Torch plugin
rendered PET_NPROC_PER_NODE=-5 and set the trainer PodSet count to 0),
failing at pod runtime instead of at admission.

Add +kubebuilder:validation:Minimum=0 to numNodes and an XValidation
self >= 1 rule to numProcPerNode, and regenerate the CRDs so the API
server rejects invalid values at admission. numNodes allows 0 to keep the
API aligned with JobSet's replicas (kubernetes-sigs/jobset#1279), rejecting
only negatives. numProcPerNode stays >= 1 since zero processes-per-node is
meaningless; it uses the same XValidation form as the existing
FluxMLPolicySource.NumProcPerNode contract so the kube-api-linter keeps the
field a pointer (nil means "auto" for the Torch runtime, which a plain
Minimum=1 marker would let the linter strip).

Apply the same XValidation self >= 1 rule to
MPIMLPolicySource.NumProcPerNode so the MPI MLPolicy is validated
consistently with the Flux MLPolicy and the TrainJob trainer field. For
MPI the value is written straight into the OpenMPI hostfile as the slots
count and into OMPI_MCA_orte_set_default_slots, so a zero or negative
value produced an invalid hostfile at pod runtime.

Drop the numProcPerNode < 1 check from the Flux plugin webhook now that
the CRD schema rejects the value earlier at admission, and remove the two
plugin unit test cases that covered it. A pre-existing Flux
numProcPerNode < 1 integration case expected a Forbidden error from the
plugin webhook; the value is now rejected by the CRD schema, so its
matcher is updated to Invalid.

The OptimizationJob CRD embeds the same Trainer type, so its generated
schema picks up both markers as well.

Closes #3805

Signed-off-by: Sanskar Singh <sanskarsinghty1234@gmail.com>

1 of 1 new or added line in 1 file covered. (100.0%)

2568 of 3742 relevant lines covered (68.63%)

0.8 hits per line