wwu-mmll / confound_corrected_cpm / 35615129980

88%

main: 88%

Build Type

push

github

Committed by NilsWinter

Commit Message

Fix: permutation test reported undefined models as maximally significant

Found by Nils immediately after Phase 9 landed. Running without covariates
still produced p-values for every model, and not harmless ones: the
covariates/full/residuals/increment rows all came out at exactly
1/(n_perms+1) -- the permutation floor, the most significant p-value the test
can report -- for models that do not exist.

The mechanism is the same one Phase 9 fixed in the classification metrics, in
a different file. Those rows are NaN placeholders in cv_results_summary.csv.
Every comparison against NaN is False, so the count of permutations beating the
observed value came out 0, and the +1 correction turned that into the smallest
possible p-value instead of no p-value at all.

_calculate_group_p_value now returns NaN when the observed statistic or the
whole null is undefined. p_values.csv keeps the same shape as
cv_results_summary.csv -- NaN placeholders, consistent with fill-don't-reshape
-- and ReportDataLoader.load_p_values / load_permutations filter them out for
the report, like the other loaders already did.

Root cause of my missing it: two paths read the same results. I filtered the
one through ReportDataLoader and left PermutationManager, which reads
cv_results_summary.csv directly via ResultsManager.load_cv_results. That is the
recurring shape of every silent bug in this package.

Also fixes a fragile assertion I wrote in the Phase 9 tests: it searched the
raw report HTML for "nan", which matches inside the base64 of any embedded
figure. It now strips data URIs first.

Three regression tests, all verified to fail against the unfixed code.
Suite 312 -> 315.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Coverage Stats

10 of 10 new or added lines in 2 files covered. (100.0%)

83 existing lines in 9 files now uncovered.

2503 of 2864 relevant lines covered (87.4%)

0.87 hits per line