sandialabs / LoQS / 37851021280

89%

push

github

Fix the checkpoint lock-contention race with SWMR progress ledgers (#132) (#142)

* Add standalone SWMR progress ledger and harden worker_id()

Introduces the single-writer/multi-reader progress-ledger primitives that later stages will wire into item- and shot-level checkpointing.
- New loqs/internal/swmrledger.py: configurable ledger fields, auto-extending done/wall_clock_times datasets, in-flight/heartbeat tracking, and a SwmrLedgerSnapshot read-back.
- worker_id() now appends a UUID-derived suffix and caches its value per-process, gated on os.getpid() so a cached value is never reused across a fork boundary.
- Adds tests/internal/test_swmrledger.py with single-process unit tests and real multi-process concurrency tests proving cross-process visibility and lock-free access.

* Migrate MultiProgramRunner's item checkpointing onto the SWMR ledger

Wires MultiProgramRunner's worker checkpoint writes onto the SWMR ledger primitives added in the prior commit, replacing retry-wrapped shared-file writes with a per-process cached ledger plus one-file-per-item payloads.
- _get_worker_ledger lazily opens and caches each worker's SWMR ledger handle per checkpoint directory for the life of the process.
- _write_item_checkpoint_with_ledger writes an item's payload via h5py.File's own with statement, then marks it done in the ledger only once that block exits -- the ordering a reader relies on to safely open a payload file the instant it sees done.
- _write_current_item_index_with_retry/_write_item_checkpoint_entries_with_retry and their now-obsolete retry-budget tests are removed; _generic_chunk_worker/_run_serial/_process_and_checkpoint_item are rewired onto the new functions.

* Read and consolidate MultiProgramRunner's SWMR ledger and payload checkpoints

Update MultiProgramRunner's resume, consolidation, and live-polling paths to read the SWMR ledger and per-item payload files written by its checkpoint writer, while staying backward-compatible with pre-exist... (continued)

3512 of 4134 branches covered (84.95%)

Branch coverage included in aggregate %.

11078 of 12074 relevant lines covered (91.75%)

0.92 hits per line