Add Unit Conformance Policy and Adherence Rules - #741
Conversation
Eleven silent failures in crop_extent handling produced wrong answers with no message: wrapped spellings gave empty grids, sub-cell and non-overlapping crops gave the full file, a cropped type-4 read reported the full-file extent, and buffer was dropped for every descriptor-cropped NetCDF file. Adds the first Fortran regression coverage for the descriptor-crop path and documents what a Topography represents across the antimeridian. Signed-off-by: Kyle Mandli <kyle.mandli@gmail.com> Assisted-by: claude claude-opus-5[1m]
A URL in topofiles was mangled by os.path.abspath into a bogus local path before the reader's existing URL guard could see it; it is now rejected with the fetch_remote_topo recipe. A crop crossing the antimeridian raised 'crop_bounds exceed file extent' even though _compute_lon_entries already covered it; TopographyData.write now resolves entries before writing and splits such a crop into one descriptor entry per side, carrying buffer and coarsen onto each so the Fortran buffer fix applies. Adds byte-exact topo.data goldens and the first end-to-end test of two entries with differing lon_wrap_offset. Signed-off-by: Kyle Mandli <kyle.mandli@gmail.com> Assisted-by: claude claude-opus-5[1m]
GeoClaw's units policy was implemented in the NetCDF readers but written down nowhere, so enforcement drifted. Adds dev/design/units_policy.md, a UNITS_POLICY registry that both the doc table and a conformance test are generated from, and fixes four violations: CSVFault.read parsed unit annotations from column headings and discarded them (alaska1964.csv read as Mw 5.20 instead of 8.53), input_units was a mutated mutable default that silently declared SI, a unit-less dtopo time axis was silently assumed to be seconds, and two docstrings claimed GeoClaw does not convert on read while it does. Non-conforming rows are xfail(strict) so the remaining ASCII gaps stay visible. Signed-off-by: Kyle Mandli <kyle.mandli@gmail.com> Assisted-by: claude claude-opus-5[1m]
Was not clear that the ASCII files have assumed units and needed some clarification as to how the proposed unit rules apply there.
PR Detailed Description -- Summarized by ClaudeBranch: The accountability mechanismA table in a document is exactly the kind of thing that quietly stops being
Verified both directions rather than assumed:
Rows that do not yet meet the policy carry a Before and afterRunning the conformance suite before any fix: 11 passed, 5 xfailed, 1 failed.
The four fixes1.
|
| before | after | |
|---|---|---|
subfaults[0].depth |
17.94 |
17940.0 |
Mw |
5.20 | 8.53 |
The 1964 Alaska earthquake read as a moderate quake. It survived because the
only test of CSVFault.read passes input_units explicitly and never exercises
the heading path — it documented the working route and never asked whether the
advertised one worked.
Precedence, now handled in one place (_resolve_input_units): an explicit
input_units entry always wins; the heading fills in only columns the caller did
not specify; a disagreement warns rather than being resolved quietly. Explicit
beats implicit so a caller who knows better than a bad heading can say so.
2. input_units={} — a mutable default, mutated
Two problems, and they had to land together: fixing #1 activates this one.
CSVFault.readwroteinput_units[name] = unitsinto the shared default
dict, so units parsed from one file would leak into the next call that
omitted the argument. The caller's dict is now never mutated.- Omitting the argument set
standard_units(m/m/m/m/Pa) silently — rule 1
inverted.Nonenow means "not specified" and warns, naming the assumed
units. An explicit{}means "my data really is SI" and stays silent: the
deliberate escape hatch.
Deliberately not warned on: Fault() with no subfaults and no units, since
constructing an empty fault and filling it later converts nothing.
3. dtopo numeric time axis assumed seconds, silently
The no-units branch of DTopoInspector._compute_time_axis was a bare
seconds = arr. A file in hours is read 3600× too fast. The assumption stays —
it is the long-standing contract and files rely on it — but it now says so. This
was the one NetCDF path breaking rule 1, in a module that states the rule at
netcdf_utils.py:146 ("callers must never silently assume seconds").
4. Docstrings claiming the opposite of rule 3
Topography.read said "GeoClaw does not convert on read; pre-convert non-meter
data to meters first" and promised a ValueError, while topotools.py
applies the factor and netcdf_utils.py warns "converting to 'm' on read".
Same claim in fetch_remote_topo. Not merely stale — actively harmful, telling
users to pre-convert data that needs no pre-converting and promising an
exception that never comes. Pinned by a test so it cannot creep back.
Also fixed
Topography.read(nc_params=None) crashed with AttributeError: 'NoneType' object has no attribute 'get' — None is the natural "no options" value, and
nc_params={} was another shared mutable default. Found by the conformance
fixtures.
Verification
- Every fix confirmed to fail on revert, each with its own failure mode:
DID NOT WARN(×2),assert 'm' == 'km', and the stale docstring claim. The
rows that were already conforming stayed green, which is the check that this
did not disturb them. test_read_csv_make_dtopopasses unchanged. It suppliesinput_units
agreeing with the heading, so the fix must be a no-op for it — the check that
the precedence rule is right.- End-to-end behaviours confirmed by hand: heading honoured (
Mw 8.53); nothing
specified → warns; explicit beats a disagreeing heading and says so; caller's
dict unmutated. - 561 passed / 4 skipped / 3 xfailed; 12 regression tests;
met_forcing,
topo_crop,topo.datagoldens and the*_test_data.tt3dtopo baselines all
byte-identical.
Release note — behaviour change
A CSV subfault file whose column headings annotate units (Depth(km)) now has
those units applied. Anyone relying on the previous behaviour was getting
wrong numbers, by orders of magnitude in the worst case; their results will
change. Omitting input_units entirely now emits a warning — pass
input_units={} to state that the data really is in SI and silence it.
Follow-up: PR B2
The two ASCII rows are visible in the generated table as not conforming, and
their tests xfail. B2 closes them and unifies the override:
assume_unitson ASCII topo/dtopo reads, and_check_magnitudeextended to
ASCII (raising, as on the NetCDF side;skip_sanity_checkis the opt-out).assume_unitsacceptingstr | dictacross the NetCDF inspectors, with
MetInspector'sboolkept working behind a deprecation. Itsboolis
defensible, not an oversight — Met covers several variables with different
contract units, andformat_units={role: unit}is its per-role string form —
and the design doc now says so.- Case-insensitive unit matching (
MetersandKMcurrently raise) and a
conversion for feet, which has no entry at all, so a file that correctly
declaresunits="ft"cannot be read.
GeoClaw has a units policy that is not stated anywhere implemented implicitly in a bunch of places and has override and warning behavior that varies by file and input type. For instance, the netCDF readers behave differently than ASCII files primarily because netCDF tends to be self-describing. Because of the newer development of topo, dtopo and met forcing this has drifted further apart.
This PR writes the policy down, adds tests, and fixes 4 places where the policy was violated. It adds not API capabilities, this just ensures the code does what we said it does.
The rules, reconstructed from the code and now stated in
dev/design/units_policy.md:Note that ASCII files can clearly not declare units, so (1) is not applied. Overriding of the assumed units in that case are applied and (5) is applied.