Skip to content

test: fix test_dunedin_pace_normalization on pandas 3.0 (fixture orientation)#210

Merged
marcbal77 merged 1 commit into
bio-learn:masterfrom
IsraelAfangideh:fix/dunedin-pace-test-fixture-orientation
Jul 11, 2026
Merged

test: fix test_dunedin_pace_normalization on pandas 3.0 (fixture orientation)#210
marcbal77 merged 1 commit into
bio-learn:masterfrom
IsraelAfangideh:fix/dunedin-pace-test-fixture-orientation

Conversation

@IsraelAfangideh

Copy link
Copy Markdown
Contributor

Summary

Fixes #209.

test_dunedin_pace_normalization fails on pandas 3.0 with
KeyError: 'cg00000029'. The DunedinPACE normalization output is correct
the problem is that the test compares against a reference fixture in a different
axis orientation, and the comparison logic depended on stack()'s NaN handling.

Root cause

  • actual (live loader output) is CpG-indexed (index = cg00000029, …;
    columns = sample GSMs).
  • expected (testset/pace_normalized.pkl) stores probe IDs in an ID_REF
    column
    with a default integer RangeIndex.

The axes never aligned. On older pandas, DataFrame.stack() silently dropped
the misaligned NaNs, so actual[mask].stack() was empty and the test passed
without ever comparing values. On pandas 3.0 the NaNs are kept, so the
diagnostic loop runs and does a label-based .at[('cg00000029', …)] lookup
against the RangeIndex, raising KeyError. (This only surfaces once the
read-only-array fix from #206 lets normalization run to completion; before that,
it aborted earlier with a ValueError.)

Fix

  • Align the fixture's orientation to actual before comparing (promote ID_REF
    to the index and reindex to actual's axes).
  • Count mismatches from the boolean mask directly, so the result no longer
    depends on version-specific stack() NaN handling.

The normalization computation is untouched — after realignment the values match
the fixture to ~5e-16 (0 of 200000 cells exceed the 1e-6 tolerance).

Testing

  • pytest biolearn/test/test_model.py::test_dunedin_pace_normalization — now
    passes on Python 3.11 / pandas 3.0.3 / numpy 2.4.6 (previously errored).
  • Full test_model.py + test_dunedin_pace.py green on the same stack, apart
    from pre-existing, unrelated test_models[GPAge*] failures (missing optional
    GPy dependency in my local env).
  • black --check clean.

Note

This is the test-side fix I offered in #209, chosen because it's fully reviewable
in plain code and keeps the existing fixture. If you'd rather regenerate
pace_normalized.pkl
in the current CpG-indexed orientation instead, happy to
switch to that — just let me know.

…ntation)

The reference fixture (pace_normalized.pkl) stores probe IDs in an ID_REF
column with a default RangeIndex, while the loader returns CpG-indexed data.
The two axes never aligned; on older pandas DataFrame.stack() silently dropped
the misaligned NaNs, so the test passed without ever comparing values. On
pandas 3.0 the NaNs are kept, surfacing a KeyError from a label-based .at[]
lookup against the RangeIndex.

Align the fixture's orientation to the actual output before comparing, and
count mismatches from the boolean mask directly so the result no longer depends
on version-specific stack() NaN handling. The normalization output itself is
unchanged (values match the fixture to ~5e-16).

Fixes bio-learn#209.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@marcbal77

Copy link
Copy Markdown
Member

Confirmed the fix locally on pandas 2.2.3 too. The realignment is genuine, 20000/20000 index overlap, 0 NaNs, max diff ~5.5e-16 across all 200k cells, so the values are actually being compared now instead of the old vacuous pass. Merging. This also covers #207. Picking up #208 next. Thanks for the contribution.

@marcbal77 marcbal77 merged commit 0d714f5 into bio-learn:master Jul 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test_dunedin_pace_normalization fails on pandas 3.0 — stale pace_normalized.pkl fixture orientation (normalization output is correct)

2 participants