You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
where the equivalent netCDF attribute would show Signal Selection Source Fraction 3. On the two committed ATL06 structural fixtures there are 4232
such occurrences.
Beyond being ugly, it can manufacture a false difference: if two files store the
same attribute under different HDF5 string types (File A fixed-length, File B
variable-length), the values render as b'NASA' and NASA and are counted as a
difference. Confirmed by building exactly that pair — before decoding they are
reported as differing; with decoding the comparison returns a total of 0.
#77 (PR #344, merged) fixed this for global (root-level) attributes only.
Why the existing decode doesn't cover this
_value_to_comparable_str in getters.py decodes bytes correctly, and root
attributes reach it as raw h5py values. Variable attributes do not: the HDF5
branch of Comparison._create_var_properties stringifies eagerly as it reads,
so by the time get_attribute_value_as_str → _value_to_comparable_str sees the
value it is already the string"b'NASA'". isinstance(value, bytes) is False
and the decode is a no-op. The fix therefore belongs in _create_var_properties, not in _value_to_comparable_str.
Proposed fix
Stop stringifying at read time, or decode before doing so, in the hdf5 branch of _create_var_properties. Note there are three str() calls in that branch, not
one — the scalar case (:563), the list-of-references case (:556), and the IndexError fallback (:560).
This is more delicate than the root-attribute change: the same branch resolves h5py
object references via __name_from_h5_ref, and h5py.ref_dtype values must keep
going down that path. The str() cannot simply be dropped. The cleanest shape is
probably to hand the raw value to _value_to_comparable_str for the non-reference
cases and leave reference handling untouched.
Verification
A cross-string-type regression test: two HDF5 files storing the same variable
attribute as fixed-length and variable-length strings should compare equal (this
is the false-difference case above; the root-attribute equivalent is test_get_root_attributes_hdf5_fixed_length_string_matches_netcdf).
Summary
Variable-level attributes read from HDF5 files are rendered as Python byte reprs
rather than text. An ATL06 comparison prints
where the equivalent netCDF attribute would show
Signal Selection Source Fraction 3. On the two committed ATL06 structural fixtures there are 4232such occurrences.
Beyond being ugly, it can manufacture a false difference: if two files store the
same attribute under different HDF5 string types (File A fixed-length, File B
variable-length), the values render as
b'NASA'andNASAand are counted as adifference. Confirmed by building exactly that pair — before decoding they are
reported as differing; with decoding the comparison returns a total of 0.
#77 (PR #344, merged) fixed this for global (root-level) attributes only.
Why the existing decode doesn't cover this
_value_to_comparable_stringetters.pydecodesbytescorrectly, and rootattributes reach it as raw h5py values. Variable attributes do not: the HDF5
branch of
Comparison._create_var_propertiesstringifies eagerly as it reads,so by the time
get_attribute_value_as_str→_value_to_comparable_strsees thevalue it is already the string
"b'NASA'".isinstance(value, bytes)is Falseand the decode is a no-op. The fix therefore belongs in
_create_var_properties, not in_value_to_comparable_str.Proposed fix
Stop stringifying at read time, or decode before doing so, in the
hdf5branch of_create_var_properties. Note there are threestr()calls in that branch, notone — the scalar case (
:563), the list-of-references case (:556), and theIndexErrorfallback (:560).This is more delicate than the root-attribute change: the same branch resolves h5py
object references via
__name_from_h5_ref, andh5py.ref_dtypevalues must keepgoing down that path. The
str()cannot simply be dropped. The cleanest shape isprobably to hand the raw value to
_value_to_comparable_strfor the non-referencecases and leave reference handling untouched.
Verification
attribute as fixed-length and variable-length strings should compare equal (this
is the false-difference case above; the root-attribute equivalent is
test_get_root_attributes_hdf5_fixed_length_string_matches_netcdf).EXPECTED_ATL06_DIFFERENCESwill move again. It is 4978 as of Add global attributes to comparison display #344,merged in
c2a18b4. Report the number the suite actually produces and state the arithmetic,as with Add global attributes to comparison display #344.
strfor text attributes — worth confirming rather than assuming.
Context
Split out of #77 / PR #344 deliberately, to keep that PR's scope to root-level
attributes and avoid moving the pinned ATL06 count twice in one change.