kubeflow / trainer / 22885491032

push

github

feat: support for flux framework as hpc manager (#3188)

* feat: support for flux framework as hpc manager
Flux supports the majority of MPI flavors/variants, and can be used
to bootstrap MPI as a plugin. It adds other features for scheduling
and topology that can be used for simulations and ai/ml jobs.
This changeset adds the plugin implementation, including the
plugin module, tests, and an example with a small README to
serve as documentation for the time being.

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

* review: slim down use of annotations

We still need to put the original command in an annotation
to retieve later, but others can be re-derived from the
environment

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

* review: clean up tasks and remove extra FluxPolicySource

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

* review: ensure to unset trainer args

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

* review: better support for nproc per node and nodes

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

* review: add constants types for flux

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

* review: add gpu specification from resources

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

* Fix TrainJob NumProcPerNode

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Remove elasticPolicy from kubebuilder:validation

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* chore: use go embed for long string templates flux plugin

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

* cleanup: removed un-used annotation

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

---------

Signed-off-by: vsoch <vsoch@users.noreply.github.com>
Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
Co-authored-by: vsoch <vsoch@users.noreply.github.com>
Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

268 of 353 new or added lines in 3 files covered. (75.92%)

1654 of 2833 relevant lines covered (58.38%)

0.68 hits per line