pirl-unc / hitlist / 26975890813
77%

DEFAULT BRANCH: main
Ran
Jobs 1
Files 29
Run time 1min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 76.832%. Remained the same
26975890813

push

github

web-flow 
v1.30.60: share full_observations_df via Arrow IPC mmap, not pickle (#262) (#271)

* v1.30.60: share full_observations_df via Arrow IPC mmap, not pickle (#262)

The full_observations_df xdist fixture was shared via an on-disk pickle:
one worker built it, the rest pickle.load'd — but pickle.load
deserializes into each worker's PRIVATE heap, so N workers held N copies
of the ~4.4M-row table (~4 GB RSS each per #244's vmmap).

Switch to Arrow IPC (Feather v2) + memory_map:

- New tests/xdist_cache.py::load_or_build_mmapped_arrow mirrors the
  pickled helper (same flock + atomic temp+rename), but writes Arrow IPC
  and reads it back via pa.memory_map + ipc.open_file.
- to_pandas(split_blocks=True, zero_copy_only=False) keeps numeric / bool
  / dictionary (categorical) columns zero-copy — each column wraps the
  shared mmap buffer instead of being consolidated into a private block.
  The OS unified buffer cache holds those pages ONCE across all workers.
- conftest full_observations_df now uses it (.pkl -> .arrow filename).

Pairs with #263: the metadata columns made categorical there round-trip
as Arrow dictionary buffers, so they mmap-share too. Object/string
columns (peptide, etc.) still materialize per worker — unavoidable
without consuming Arrow directly.

Verified: 20 integration tests pass under -n auto with the mmap fixture;
mmap-read frames preserve dtypes and survive source-handle GC (zero-copy
buffers keep the mapping alive via pyarrow refcounting). load read ~0.25s
vs pickle's seconds. load_or_build_pickled retained for other fixtures.

* Address review: builder worker round-trips too (xdist consistency)

Previously the builder worker returned its raw in-memory frame (writable,
RangeIndex) while reader workers returned the mmap read-back (read-only,
Int64 index). Since xdist assigns each test to one worker
nondeterministically, a test could behave differently depending on which
worker it landed on. Now the builder also returns via the mmap read ... (continued)

4729 of 6155 relevant lines covered (76.83%)

0.77 hits per line

Jobs
ID Job ID Ran Files Coverage
1 26975890813.1 29
76.83
 GitHub Action Run
Source Files on build 26975890813