opendatahub-io / kubeflow-sdk / 22902595383

76%

Build Type

push

github

Committed by web-flow

Commit Message

fix: resolve distributed JIT checkpoint intermittent resume failure (#86)

* fix: all ranks write sentinel file

* fix: add barrier before JIT checkpointing to avoid race condition and ensure all ranks save at the same step

* fix: make jit checkpoint thread non-deamon and wait for thread completion

* fix: address coderabbit comment. Added timeout for thread join.

* chore: update warning message

* fix: move thread.join() to on_step_end() callback.

* fix: race condition in JIT checkpoint by making it synchronous.

* fix: after JIT checkpoint ensure to gracefully exit, so avoid user save_model() code running.

* test: update tests

* chore: rebase final model upload changes

* fix: deadlock issues when checkpoint is requested while periodic checkpoint in progress.

Run Details

95 of 162 new or added lines in 2 files covered. (58.64%)

4 existing lines in 1 file now uncovered.

5080 of 7378 relevant lines covered (68.85%)

1.38 hits per line