opendatahub-io / kubeflow-sdk / 22678882394

76%

Build Type

push

github

Committed by web-flow

Commit Message

fix: distributed checkpointing issues with PVC and S3 (#83)

* fix: address review comments from PR #80

* chore: update print statements for better logging for ease during debugging in distributed training settings.

* fix: race condition to delete incomplete checkpoint

- In shared RWX PVCs only global rank should delete upon resume of training.
- Update async JIT checkpoint with better mechanism to identify local rank 0. The old method was brittle.

* fix: create checkpoint markers at enqueue time with retries

 - Write per-rank incomplete markers during on_save to avoid resume races
 - Retry marker create/delete with small backoff

* fix: review comments

* fix: add retry mechanism for S3 access verification on startup in transformerTrainer init()

* fix: coderabbit comment

Run Details

18 of 72 new or added lines in 2 files covered. (25.0%)

3 existing lines in 1 file now uncovered.

4825 of 7057 relevant lines covered (68.37%)

1.37 hits per line