kubeflow / trainer / 26168404895

62%

Build Type

push

github

Committed by web-flow

Commit Message

fix(runtimes): add validation for LoRA multi-node and immutable trainer args (#3302)

* fix(torchtune): add validation for LoRA multi-node and immutable args

Add two missing validations to validateTorchTune:

1. Reject LoRA fine-tuning when numNodes > 1. getRecipeAndConfig
   silently falls through to full_finetune_distributed in this case,
   discarding LoRA args without any user-visible error.

2. Reject immutable runtime configs (output_dir, tokenizer.path,
   checkpointer.checkpoint_dir, tokenizer.merges_file) in
   spec.trainer.args. These are injected by the runtime via
   extractOverridesFromRuntime and must not be overridden by the user.

Resolves the TODO added by @Electronic-Waste in torchtune.go.

Signed-off-by: Krish Garg <gargkrish06@gmail.com>
Signed-off-by: krishdef7 <gargkrish06@gmail.com>

* fix(runtimes): add validation for LoRA/QLoRA multi-node and immutable trainer args

Signed-off-by: krishdef7 <gargkrish06@gmail.com>

---------

Signed-off-by: Krish Garg <gargkrish06@gmail.com>
Signed-off-by: krishdef7 <gargkrish06@gmail.com>

Coverage Stats

17 of 17 new or added lines in 1 file covered. (100.0%)

3 existing lines in 1 file now uncovered.

2205 of 3532 relevant lines covered (62.43%)

0.72 hits per line