Skip to content

Add global attributes to comparison display - #344

Merged
danielfromearth merged 5 commits into
nasa:developfrom
agneay:feature/issue-77
Sep 4, 2026
Merged

Add global attributes to comparison display#344
danielfromearth merged 5 commits into
nasa:developfrom
agneay:feature/issue-77

Conversation

@agneay

@agneay agneay commented Aug 22, 2026

Copy link
Copy Markdown

GitHub Issue: #77

Description

Adds global (root-level) file attributes to the comparison report. Previously, ncompare only showed per-variable attributes when --show-attributes was set; attributes attached to the file itself (e.g. title, history, source) — not associated with any variable — were never displayed or diffed.

This adds:

  • get_root_attributes() in ncompare/getters.py, which reads global attributes from either a netCDF4 Dataset or an h5py.File, following the same file-opening pattern already used by get_root_dims() / get_root_groups().
  • Comparison._print_root_attributes() in ncompare/Comparison.py, which diffs and prints those attributes side-by-side (same format as the existing "Root-level Dimensions" / "Root-level Groups" sections), and folds differences into the existing num_attribute_diffs tally alongside variable-level attribute diffs. This is gated by the existing show_attributes flag, matching how variable attributes are already gated.

Local test steps

  • Added test_get_root_attributes_netcdf and test_get_root_attributes_hdf5 in tests/test_getters.py, unit-testing the new getter against synthetic in-memory netCDF and HDF5 files with known global attributes.
  • Added test_root_attributes_included_when_show_tributes_excluded_when_not_show_attributes intests/test_core.py, verifying the new section appears/disappears correctly based on the show_attributes flag and that a
    genuine attribute difference is reflected in the
  • Ran the full existing test suite locally (pytest -q): 26 passed. The 3 pre-existing failures (test_console_version,
    test_console_help, test_full_run_to_csv_outputmissing ncompareonPATHin a fresh venv; apre-existing Windows-only newline bug inwrite_history_to_csv` unrelated to this change) and were failing identically before
    this PR.
  • Ran ruff check and ruff format --check on all changed files — clean.
  • Ran mypy on the changed source files — no newmissing-stubs notice, unrelated to this change).

Overview of integration done

Ran a full end-to-end comparison against the repodata/test_a.ncvs.tests/data/test_b.nc), whichcarry real differing global attributes (historytimestamps differ;descriptionandsourcematch). Confirmed the new "Root-level Attributes:" section correctly reportas shared andhistoryas a difference, and thatthe summary counts and "Differences were found in these attributes" list update consistently (33 shared / 14+23 non-shared,history added to the difference-types list). Re_test_golden_file.txt and .xlsx goldenregression fixtures to reflect this new output; the diff against the previous golden files contains only the new section plus the
corresponding count updates — no unrelated drift.

PR Acceptance Checklist

  • Unit tests added/updated and passing.
  • Integration testing
  • CHANGELOG.md updated

📚 Documentation preview 📚: https://ncompare--344.org.readthedocs.build/en/344/

@danielfromearth danielfromearth added the enhancement New feature or request label Aug 30, 2026

@sylvesterkaczmarek sylvesterkaczmarek left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get_root_attributes() stringifies HDF5 fixed-length strings as b'...', while the equivalent NetCDF attribute becomes plain text. An HDF5 np.bytes_('NASA') and NetCDF "NASA" therefore produce a false root-attribute difference. Decode/normalise HDF5 string attributes before comparison and add a cross-format regression for fixed-length text.

@danielfromearth
danielfromearth changed the base branch from main to develop September 3, 2026 21:33
@danielfromearth

Copy link
Copy Markdown
Collaborator

Thank you @agneay, this is a really nice contribution! 🎉🎉 #77 has been sitting open for a long time and it's great to have a proposal to resolve it!

A few things to sort out before it's ready to merge, mostly because the PR was written to target main, which is a ways behind develop now. So, I've retargeted this to develop.

Could you please merge develop in? Your source changes merge cleanly. Three files conflict: CHANGELOG.md, tests/test_core.py, tests/test_getters.py. Resolving those should be quick (and if you don't feel comfortable doing that; I could resolve those).

It looks like EXPECTED_ATL06_DIFFERENCES will need to become 4978 (from 4958) in tests/test_core.py, or else test_icesat_structure fails. I merged your branch onto develop locally, to get that new number. You could update the PR description too: I think the two ATL06 fixtures share 48 root attribute names, 10 differ in value, and _total_difference_count() counts a "both" difference on each side; hence +20.

Two code suggestions:

  • _attribute_as_str overlaps get_attribute_value_as_str (getters.py:43), which already stringifies values and truncates long iterables to five elements. Since yours doesn't, a long global attribute prints unbounded while the variable-level equivalent gets cut off. Could you factor out the shared value-level part and call it from both? Keep your bytes decode, just apply it first (bytes is iterable, so I think if you did it in the other order, it would turn b"NASA" into [78, 65, 83, 65]).
  • _print_root_attributes duplicates the six tally lines from _var_attribute_side_by_side, which is a closure and so can't be called from outside. Promoting it to a small private method and using it in both places would stop the two drifting apart.

Smaller things:

  • please regenerate the golden files from the merged branch rather than the main-based ones (the txt/csv checks are lenient enough that a stale one can pass for the wrong reason);
  • get_root_attributes could use mode="r" and degrade to {} on error, matching how _get_hdf5_root_dims handles odd HDF5 files;
  • _print_root_attributes wants a numpy docstring and -> None;
  • and console.py's help still says --show-attributes covers "variable attributes" only.

On the CHANGELOG: can you spell out the CSV newline="" fix in its own bullet citing this PR (([#344](https://github.com/nasa/ncompare/pull/344))) instead of #77? It would be great to have your entries also end with attribution: ([**@agneay**](https://github.com/agneay))!

Bring the root-attribute feature up to date with develop and address
review feedback:

- getters: factor a shared `_value_to_comparable_str` used by both
  `get_attribute_value_as_str` and `get_root_attributes`, decoding bytes
  before the iterable-truncation branch so `b"NASA"` is not rendered as a
  list of ints; `get_root_attributes` opens read-only and returns `{}`
  when a file's attributes cannot be read, mirroring `_get_hdf5_root_dims`
- Comparison: promote the `_var_attribute_side_by_side` closure to a
  private `_tally_attribute_difference` method and call it from both the
  variable-level and root-level attribute paths; give
  `_print_root_attributes` a numpy-style docstring and `-> None`
- console: `--show-attributes` help now mentions global (root-level)
  attributes
- tests: keep develop's `get_root_dims` tests alongside the new
  `get_root_attributes` tests; bump `EXPECTED_ATL06_DIFFERENCES`
  4958 -> 4978 for the 10 shared root attributes that differ in value
  (counted on both sides by `_total_difference_count`)
- CHANGELOG: merge the entries into develop's Unreleased section; the
  Windows CSV newline fix now cites PR nasa#344

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@codecov

codecov Bot commented Sep 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.95%. Comparing base (9802b35) to head (daa0cd7).
⚠️ Report is 1 commits behind head on develop.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #344      +/-   ##
===========================================
+ Coverage    96.61%   96.95%   +0.33%     
===========================================
  Files            9        9              
  Lines          562      591      +29     
===========================================
+ Hits           543      573      +30     
+ Misses          19       18       -1     
Flag Coverage Δ
python-3.11 96.95% <100.00%> (+0.33%) ⬆️
python-3.12 96.95% <100.00%> (+0.33%) ⬆️
python-3.13 96.95% <100.00%> (+0.33%) ⬆️
python-3.14 96.92% <100.00%> (+0.34%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

… get_root_attributes

codecov flagged 4 uncovered lines in getters.py on PR nasa#344 (danielfromearth's
review): the numpy-array-of-fixed-length-strings branch in
_value_to_comparable_str, and the except/degrade-gracefully branch in
get_root_attributes. Both were reachable but untested.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFWQSRZbNCfmFLtEk8gXa4
@danielfromearth

danielfromearth commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

This is great, thank you @agneay! 🙌 It's awesome to see that getters.py is even better covered now than it was before this PR.

This is pretty much good to go. There's one tiny thing I'll tweak myself before finally merging: I'm going to narrow the "Decode HDF5 fixed-length string attributes" changelog bullet to say "global (root-level) attributes", because variable attributes still come through as b'...': That is, _create_var_properties does str(attribute_value) when it reads them (Comparison.py:563), so they're already strings before the new helper function sees them. Your change is scoped right; the bullet just sounds broader than it is. I'll open a separate issue for the variable-attribute part of that.

Thanks for sticking with this!

The decode in `_value_to_comparable_str` only reaches global (root-level)
attributes. Variable attributes are stringified as they are read, in the HDF5
branch of `_create_var_properties`, so they still render as byte reprs and the
previous wording promised more than the change delivers.
@danielfromearth
danielfromearth merged commit c2a18b4 into nasa:develop Sep 4, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants