test: fix test_dunedin_pace_normalization on pandas 3.0 (fixture orientation)#210
Merged
marcbal77 merged 1 commit intoJul 11, 2026
Conversation
…ntation) The reference fixture (pace_normalized.pkl) stores probe IDs in an ID_REF column with a default RangeIndex, while the loader returns CpG-indexed data. The two axes never aligned; on older pandas DataFrame.stack() silently dropped the misaligned NaNs, so the test passed without ever comparing values. On pandas 3.0 the NaNs are kept, surfacing a KeyError from a label-based .at[] lookup against the RangeIndex. Align the fixture's orientation to the actual output before comparing, and count mismatches from the boolean mask directly so the result no longer depends on version-specific stack() NaN handling. The normalization output itself is unchanged (values match the fixture to ~5e-16). Fixes bio-learn#209. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Member
|
Confirmed the fix locally on pandas 2.2.3 too. The realignment is genuine, 20000/20000 index overlap, 0 NaNs, max diff ~5.5e-16 across all 200k cells, so the values are actually being compared now instead of the old vacuous pass. Merging. This also covers #207. Picking up #208 next. Thanks for the contribution. |
marcbal77
approved these changes
Jul 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #209.
test_dunedin_pace_normalizationfails on pandas 3.0 withKeyError: 'cg00000029'. The DunedinPACE normalization output is correct —the problem is that the test compares against a reference fixture in a different
axis orientation, and the comparison logic depended on
stack()'s NaN handling.Root cause
actual(live loader output) is CpG-indexed (index =cg00000029, …;columns = sample GSMs).
expected(testset/pace_normalized.pkl) stores probe IDs in anID_REFcolumn with a default integer
RangeIndex.The axes never aligned. On older pandas,
DataFrame.stack()silently droppedthe misaligned NaNs, so
actual[mask].stack()was empty and the test passedwithout ever comparing values. On pandas 3.0 the NaNs are kept, so the
diagnostic loop runs and does a label-based
.at[('cg00000029', …)]lookupagainst the
RangeIndex, raisingKeyError. (This only surfaces once theread-only-array fix from #206 lets normalization run to completion; before that,
it aborted earlier with a
ValueError.)Fix
actualbefore comparing (promoteID_REFto the index and reindex to
actual's axes).depends on version-specific
stack()NaN handling.The normalization computation is untouched — after realignment the values match
the fixture to ~5e-16 (0 of 200000 cells exceed the 1e-6 tolerance).
Testing
pytest biolearn/test/test_model.py::test_dunedin_pace_normalization— nowpasses on Python 3.11 / pandas 3.0.3 / numpy 2.4.6 (previously errored).
test_model.py+test_dunedin_pace.pygreen on the same stack, apartfrom pre-existing, unrelated
test_models[GPAge*]failures (missing optionalGPydependency in my local env).black --checkclean.Note
This is the test-side fix I offered in #209, chosen because it's fully reviewable
in plain code and keeps the existing fixture. If you'd rather regenerate
pace_normalized.pklin the current CpG-indexed orientation instead, happy toswitch to that — just let me know.