pirl-unc / hitlist / 25751986801
76%

DEFAULT BRANCH: main
Ran
Jobs 1
Files 28
Run time 1min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 75.596% (+0.2%) from 75.408%
25751986801

push

github

web-flow 
v1.30.50: parallelize per-species mapping pass (closes #249) (#254)

* v1.30.50: parallelize per-species mapping pass via ProcessPoolExecutor (closes #249)

Per-species index builds in build_peptide_mappings are CPU-bound and
embarrassingly parallel — cProfile of a cold full build (#176) showed
~67% of total wall time in the per-species mapping block.  This change
runs canonicals concurrently using stdlib concurrent.futures (no new
deps) so the block scales roughly linearly with worker count up to the
memory ceiling.

Defaults to min(4, cpu_count // 2) workers — peak resident bounded by
~workers x largest-single-length-index ~ 4 x 3 GB = 12 GB on the human
mapping pass.  Override via HITLIST_BUILD_WORKERS=N (1 = sequential
fallback identical to pre-#249 behavior).

Tests: 7 new in tests/test_mappings.py covering env-var handling,
worker contract, and missing-index skip behavior.

* mappings: address PR #254 review — pre-fetch races, chunksize, context manager, tests

Five tightening changes from the self-review of the parallelism PR:

1. _prefetch_proteomes_for_workers(): warm UniProt FASTAs and pyensembl
   GTFs in the parent BEFORE dispatching tasks.  Eliminates the
   first-cold-build race where two workers concurrently call
   urllib.request.urlretrieve into the SAME .tmp path (corrupts the
   destination FASTA on shutil.move) or simultaneously trigger a
   pyensembl GTF download + index.  No-op when caches are warm.

2. chunksize=2 on pool.map(): keeps adjacent FASTA-clustered tasks on
   the same worker so the in-memory LRU benefit from #107 isn't fully
   defeated by ProcessPoolExecutor's default round-robin.  Strain
   variants of size >= 2 get the 2nd member's index from the
   same-process cache rather than re-loading from disk.

3. with ProcessPoolExecutor(...): replace the manual try/finally +
   pool.shutdown with a context manager.  Identical cleanup semantics,
   reads more cleanly.

4. Stale "Length-on-demand" comment block at the orch... (continued)

4442 of 5876 relevant lines covered (75.6%)

0.76 hits per line

Coverage Regressions

Lines Coverage ∆ File
75
41.96
 10.4% mappings.py
55
63.69
 0.61% builder.py
20
89.14
 0.86% proteome.py
Jobs
ID Job ID Ran Files Coverage
1 25751986801.1 28
75.6
 GitHub Action Run
Source Files on build 25751986801