Add global attributes to comparison display - #344
Conversation
sylvesterkaczmarek
left a comment
There was a problem hiding this comment.
get_root_attributes() stringifies HDF5 fixed-length strings as b'...', while the equivalent NetCDF attribute becomes plain text. An HDF5 np.bytes_('NASA') and NetCDF "NASA" therefore produce a false root-attribute difference. Decode/normalise HDF5 string attributes before comparison and add a cross-format regression for fixed-length text.
|
Thank you @agneay, this is a really nice contribution! 🎉🎉 #77 has been sitting open for a long time and it's great to have a proposal to resolve it! A few things to sort out before it's ready to merge, mostly because the PR was written to target Could you please merge It looks like Two code suggestions:
Smaller things:
On the CHANGELOG: can you spell out the CSV |
Bring the root-attribute feature up to date with develop and address
review feedback:
- getters: factor a shared `_value_to_comparable_str` used by both
`get_attribute_value_as_str` and `get_root_attributes`, decoding bytes
before the iterable-truncation branch so `b"NASA"` is not rendered as a
list of ints; `get_root_attributes` opens read-only and returns `{}`
when a file's attributes cannot be read, mirroring `_get_hdf5_root_dims`
- Comparison: promote the `_var_attribute_side_by_side` closure to a
private `_tally_attribute_difference` method and call it from both the
variable-level and root-level attribute paths; give
`_print_root_attributes` a numpy-style docstring and `-> None`
- console: `--show-attributes` help now mentions global (root-level)
attributes
- tests: keep develop's `get_root_dims` tests alongside the new
`get_root_attributes` tests; bump `EXPECTED_ATL06_DIFFERENCES`
4958 -> 4978 for the 10 shared root attributes that differ in value
(counted on both sides by `_total_difference_count`)
- CHANGELOG: merge the entries into develop's Unreleased section; the
Windows CSV newline fix now cites PR nasa#344
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #344 +/- ##
===========================================
+ Coverage 96.61% 96.95% +0.33%
===========================================
Files 9 9
Lines 562 591 +29
===========================================
+ Hits 543 573 +30
+ Misses 19 18 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
… get_root_attributes codecov flagged 4 uncovered lines in getters.py on PR nasa#344 (danielfromearth's review): the numpy-array-of-fixed-length-strings branch in _value_to_comparable_str, and the except/degrade-gracefully branch in get_root_attributes. Both were reachable but untested. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MFWQSRZbNCfmFLtEk8gXa4
|
This is great, thank you @agneay! 🙌 It's awesome to see that This is pretty much good to go. There's one tiny thing I'll tweak myself before finally merging: I'm going to narrow the "Decode HDF5 fixed-length string attributes" changelog bullet to say "global (root-level) attributes", because variable attributes still come through as Thanks for sticking with this! |
The decode in `_value_to_comparable_str` only reaches global (root-level) attributes. Variable attributes are stringified as they are read, in the HDF5 branch of `_create_var_properties`, so they still render as byte reprs and the previous wording promised more than the change delivers.
GitHub Issue: #77
Description
Adds global (root-level) file attributes to the comparison report. Previously,
ncompareonly showed per-variable attributes when--show-attributeswas set; attributes attached to the file itself (e.g.title,history,source) — not associated with any variable — were never displayed or diffed.This adds:
get_root_attributes()inncompare/getters.py, which reads global attributes from either a netCDF4Datasetor anh5py.File, following the same file-opening pattern already used byget_root_dims()/get_root_groups().Comparison._print_root_attributes()inncompare/Comparison.py, which diffs and prints those attributes side-by-side (same format as the existing "Root-level Dimensions" / "Root-level Groups" sections), and folds differences into the existingnum_attribute_diffstally alongside variable-level attribute diffs. This is gated by the existingshow_attributesflag, matching how variable attributes are already gated.Local test steps
test_get_root_attributes_netcdfandtest_get_root_attributes_hdf5intests/test_getters.py, unit-testing the new getter against synthetic in-memory netCDF and HDF5 files with known global attributes.test_root_attributes_included_when_show_tributes_excluded_when_not_show_attributesintests/test_core.py, verifying the new section appears/disappears correctly based on theshow_attributesflag and that agenuine attribute difference is reflected in the
pytest -q): 26 passed. The 3 pre-existing failures (test_console_version,test_console_help,test_full_run_to_csv_outputmissingncompareonPATHin a fresh venv; apre-existing Windows-only newline bug inwrite_history_to_csv` unrelated to this change) and were failing identically beforethis PR.
ruff checkandruff format --checkon all changed files — clean.mypyon the changed source files — no newmissing-stubs notice, unrelated to this change).Overview of integration done
Ran a full end-to-end comparison against the repodata/test_a.nc
vs.tests/data/test_b.nc), whichcarry real differing global attributes (historytimestamps differ;descriptionandsourcematch). Confirmed the new "Root-level Attributes:" section correctly reportas shared andhistoryas a difference, and thatthe summary counts and "Differences were found in these attributes" list update consistently (33 shared / 14+23 non-shared,historyadded to the difference-types list). Re_test_golden_file.txtand.xlsxgoldenregression fixtures to reflect this new output; the diff against the previous golden files contains only the new section plus thecorresponding count updates — no unrelated drift.
PR Acceptance Checklist
CHANGELOG.mdupdated📚 Documentation preview 📚: https://ncompare--344.org.readthedocs.build/en/344/