Skip to content

Bug: HDF5 variable attributes display as byte reprs (b'NASA') in the comparison report #370

Description

@danielfromearth

Summary

Variable-level attributes read from HDF5 files are rendered as Python byte reprs
rather than text. An ATL06 comparison prints

long_name:   b'Signal Selection Source Fraction 3'
source:      b'ATL06 ATBD - Q/A Statistics'
units:       b'1'

where the equivalent netCDF attribute would show Signal Selection Source Fraction 3. On the two committed ATL06 structural fixtures there are 4232
such occurrences.

Beyond being ugly, it can manufacture a false difference: if two files store the
same attribute under different HDF5 string types (File A fixed-length, File B
variable-length), the values render as b'NASA' and NASA and are counted as a
difference. Confirmed by building exactly that pair — before decoding they are
reported as differing; with decoding the comparison returns a total of 0.

#77 (PR #344, merged) fixed this for global (root-level) attributes only.

Why the existing decode doesn't cover this

_value_to_comparable_str in getters.py decodes bytes correctly, and root
attributes reach it as raw h5py values. Variable attributes do not: the HDF5
branch of Comparison._create_var_properties stringifies eagerly as it reads,

else:
    retrieved_value = str(attribute_value)   # Comparison.py:563

so by the time get_attribute_value_as_str → _value_to_comparable_str sees the
value it is already the string "b'NASA'". isinstance(value, bytes) is False
and the decode is a no-op. The fix therefore belongs in
_create_var_properties, not in _value_to_comparable_str.

Proposed fix

Stop stringifying at read time, or decode before doing so, in the hdf5 branch of
_create_var_properties. Note there are three str() calls in that branch, not
one — the scalar case (:563), the list-of-references case (:556), and the
IndexError fallback (:560).

This is more delicate than the root-attribute change: the same branch resolves h5py
object references via __name_from_h5_ref, and h5py.ref_dtype values must keep
going down that path. The str() cannot simply be dropped. The cleanest shape is
probably to hand the raw value to _value_to_comparable_str for the non-reference
cases and leave reference handling untouched.

Verification

  • A cross-string-type regression test: two HDF5 files storing the same variable
    attribute as fixed-length and variable-length strings should compare equal (this
    is the false-difference case above; the root-attribute equivalent is
    test_get_root_attributes_hdf5_fixed_length_string_matches_netcdf).
  • EXPECTED_ATL06_DIFFERENCES will move again. It is 4978 as of Add global attributes to comparison display #344,
    merged in c2a18b4. Report the number the suite actually produces and state the arithmetic,
    as with Add global attributes to comparison display #344.
  • The netCDF golden files should be unaffected, since netCDF4 already returns str
    for text attributes — worth confirming rather than assuming.

Context

Split out of #77 / PR #344 deliberately, to keep that PR's scope to root-level
attributes and avoid moving the pinned ATL06 count twice in one change.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions