Skip to content

fix(validation): compare model artifacts like-for-like (H-01) and forward t3 --subset (H-02) - #87

Merged
hydrogen1222 merged 1 commit into
mainfrom
fix/aud-008-harness-identity-and-tier-forwarding
Nov 7, 2026
Merged

hydrogen1222 merged 1 commit into
mainfrom
fix/aud-008-harness-identity-and-tier-forwarding

Conversation

@hydrogen1222

Copy link
Copy Markdown
Owner

Tracking: #61
Audit: #75 (H-01, H-02)

Why

Two defects recorded in the #75 audit ledger were fixed. Neither changes a
scientific threshold or rewrites a historical record.

H-01 — false fail on every GRACE saddle record

validation/science/scripts/neb_suite.py::model_identity_check compared

  • the engine-recorded model_sha256 (mliport.model_record.hash_model_path:
    the file's bytes, or for a directory a tree hash over
    sha256(len(rel)||rel||bytes) of every file), with
  • the manifest model_sha256, which for a directory artifact is the
    distributed archive identity (secondary_hashes verify the individual
    files separately).

For a single-file model (MACE .model, DPA .pt) those coincide. For a GRACE
SavedModel directory they can never coincide, so
metrics.model_identity.ok was false and the record was written as fail
even when the sampled barrier, CI convergence, active/band force thresholds and
the persistent negative curvature all passed.

Evidence (existing, unmodified record
validation-work/grace/t5/t5_saddle__t5h_saddle_hessian__grace.json):

status = fail
metrics.model_identity.record.model_sha256 = a8df2705…   (tree hash)
metrics.model_identity.checks.model_sha256 = false
record.model_sha256                        = d497af4c…   (manifest/archive)
barrier_forward_sampled_eV = 0.7456, ci_converged = true,
active_physical_fmax_eV_A = 0.0295 <= 0.03, min_eigenvalue ≈ -2.041 eV/Ų

H-02 — --tiers t3 could never run

run_suite.py started omat_eval.py without its required --subset, so t3
died inside the child's argparse.

What changed

  • common.artifact_sha256 — hashes a model artifact with exactly mliport's
    path-hashing algorithm, with a regression test proving it agrees with
    mliport.model_record.hash_model_path for both a file and a directory.
  • EngineContext gained an optional model_path field and an
    artifact_sha256 property that fails closed (raises) when the context has
    no model path instead of silently falling back to the manifest identity —
    that fallback is exactly the H-01 bug.
  • model_identity_check now compares like with like, and additionally records
    manifest_model_sha256 and artifact_sha256 for provenance.
  • run_suite.py gained --subset (forwarded only to omat_eval.py) and
    rejects --tiers t3 without it with an actionable message.

Tests (red → green)

tests/mliport/mliport/test_science_neb_suite.py, +6 tests:

test assertion
test_artifact_sha256_matches_mliport_path_hashing harness hash == mliport.model_record.hash_model_path for a file and a directory
test_artifact_sha256_is_order_independent_and_content_sensitive stable ordering, content sensitivity
test_model_identity_check_compares_like_with_like_for_directory_artifacts H-01 accept case: manifest b…, artifact a…, engine a… → ok is True
test_model_identity_check_still_rejects_a_genuinely_different_artifact gate not weakened: engine c… vs artifact a… → ok is False
test_run_suite_forwards_subset_to_omat_eval --tiers t3 --subset P produces omat_eval.py … --subset P
test_run_suite_rejects_t3_without_subset exit 2 and the message names --subset

The three existing saddle-workflow tests now build a real toy artifact so the
identity check runs end to end (they previously passed a fake "a"*64).

Results

pytest tests/mliport/mliport/test_science_neb_suite.py   -> 23 passed
pytest test_science_harness + test_science_neb_suite +
       test_science_md_suite + test_validation_narrative -> 115 passed
full core suite (this branch)                            -> see check runs
clean baseline (main 3d6ca2ea, pristine worktree)        -> 1688 passed, 33 skipped
ruff check + ruff format --check                          -> clean

DATA IMPACT

No stored result changed. Historical GRACE saddle records written as fail
because of this comparison are misclassified, not wrong physics: they must
be re-evaluated with the corrected harness before any scientific claim is
attached, and the old records stay untouched. No threshold was relaxed.

Not done here

AUD-007 (#82/#83) remains an open, owner-decision blocker: config-search
checkpoint restore still replaces non-representative member geometries, and NEB
geometry-level resume is still not full optimiser-state resume. Both PRs are
still Draft and are not touched by this change.

… --subset

Tracking: #61
Audit: #75 (H-01, H-02)

H-01: `neb_suite.model_identity_check` compared the engine-recorded
`model_sha256` with the validation manifest's `model_sha256`. For a
single-file artifact those are the same number, but for a directory artifact
(an exported GRACE SavedModel) the manifest value is the *distributed archive*
identity while `mliport.model_record.hash_model_path` hashes the directory
tree. The comparison could never succeed, so every GRACE saddle record was a
false `fail` even when the sampled barrier, CI convergence, active/band force
criteria and the persistent negative mode all passed.

The harness now hashes the local artifact with the same algorithm mliport uses
(`common.artifact_sha256`, a deliberate reimplementation with a test proving it
agrees with `mliport.model_record.hash_model_path` for both a file and a
directory) and records the manifest identity separately as
`manifest_model_sha256`. No scientific threshold was relaxed: a genuinely
different artifact still fails, and `resolve_profile` still verifies directory
artifacts file-by-file through `secondary_hashes` at start-up.

H-02: `run_suite.py --tiers t3` launched `omat_eval.py` without its required
`--subset`, so the tier could never produce evidence. `--subset` is now
forwarded, and `--tiers t3` without it fails immediately with an actionable
message instead of inside the child process.

Tests: +6 (artifact hashing agreement with mliport for file and directory
artifacts, order-independence/content sensitivity, the H-01 accept and reject
cases, and the H-02 forwarding plus its actionable refusal). The existing
saddle-workflow tests now build a real toy artifact so the identity check is
exercised end to end: 23 passed in test_science_neb_suite.py, 92 passed across
the science harness/neb/md/narrative suites.

DATA IMPACT: none on stored results. Historical GRACE saddle records that were
marked `fail` because of the identity comparison are *misclassified*, not
wrong physics; they must be re-evaluated with the corrected harness before any
scientific claim is attached to them, and no historical record was rewritten.

Signed-off-by: mliport agent <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant