opendatahub-io / kubeflow-sdk / 31783529735

86%

push

github

feat: add SpeculativeDecodingTrainer with Eagle3 draft model training support (#138)

* feat: Add SpeculativeDecodingTrainer with TRAIN_ONLY and DATA_ONLY mode support  (#123)

* feat: add SpeculativeDecodingTrainer for TRAIN_ONLY mode

Add SpeculativeDecodingTrainer supporting TRAIN_ONLY mode for training
speculative decoding draft models (Eagle3) using the speculators library.

- Training script injection via inspect.getsource()
- In-memory progression tracking with HTTP metrics endpoint
- PVC mount deduplication for shared storage
- Support for both pvc:// URIs and direct paths for output_dir
  and hidden_states_path
- Comprehensive validation and 45 unit tests

* fix: remove unused save_path from instrumentation functions

* feat: add DATA_ONLY mode support to SpeculativeDecodingTrainer

* remove num_nodes from SpeculativeDecodingTrainer

* fix: write termination message directly on phase completion for DATA_ONLY progression tracking

* fix: add target_layer_ids support, data_path validation, sidecar PVC check, and thread-safe termination

* feat: add AutoConfig layer detection, strict PVC validation, and HF_TOKEN forwarding to sidecar

* fix: disable trust_remote_code in AutoConfig.from_pretrained

* fix: pass HF_TOKEN to AutoConfig for gated model support

* fix: inject _REGEN_SCRIPT_B64 into rendered training script

* fix review comments: from_pretrained guard, collate dtype, text_config traversal, race condition, layer IDs format, seq_len default, double nesting

* fix: revert layer_ids_str to comma-separated to avoid double-nested JSON in sidecar

* fix: move distributed barrier outside exists check to prevent NCCL deadlock

* fix: account for completed epochs in progression percentage calculation

* fix: use per-epoch step with min cap for progression tracking

* fix: delete interrupted checkpoint when resume_from_checkpoint is enabled

* fix: only rank 0 deletes interrupted checkpoint to prevent multi-GPU race

* feat: add mandatory resour... (continued)

1288 of 1718 new or added lines in 8 files covered. (74.97%)

12927 of 15112 relevant lines covered (85.54%)

0.86 hits per line