Repository navigation
fix(validation): compare model artifacts like-for-like (H-01) and forward t3 --subset (H-02) - #87
Merged
hydrogen1222 merged 1 commit intoNov 7, 2026
Conversation
… --subset Tracking: #61 Audit: #75 (H-01, H-02) H-01: `neb_suite.model_identity_check` compared the engine-recorded `model_sha256` with the validation manifest's `model_sha256`. For a single-file artifact those are the same number, but for a directory artifact (an exported GRACE SavedModel) the manifest value is the *distributed archive* identity while `mliport.model_record.hash_model_path` hashes the directory tree. The comparison could never succeed, so every GRACE saddle record was a false `fail` even when the sampled barrier, CI convergence, active/band force criteria and the persistent negative mode all passed. The harness now hashes the local artifact with the same algorithm mliport uses (`common.artifact_sha256`, a deliberate reimplementation with a test proving it agrees with `mliport.model_record.hash_model_path` for both a file and a directory) and records the manifest identity separately as `manifest_model_sha256`. No scientific threshold was relaxed: a genuinely different artifact still fails, and `resolve_profile` still verifies directory artifacts file-by-file through `secondary_hashes` at start-up. H-02: `run_suite.py --tiers t3` launched `omat_eval.py` without its required `--subset`, so the tier could never produce evidence. `--subset` is now forwarded, and `--tiers t3` without it fails immediately with an actionable message instead of inside the child process. Tests: +6 (artifact hashing agreement with mliport for file and directory artifacts, order-independence/content sensitivity, the H-01 accept and reject cases, and the H-02 forwarding plus its actionable refusal). The existing saddle-workflow tests now build a real toy artifact so the identity check is exercised end to end: 23 passed in test_science_neb_suite.py, 92 passed across the science harness/neb/md/narrative suites. DATA IMPACT: none on stored results. Historical GRACE saddle records that were marked `fail` because of the identity comparison are *misclassified*, not wrong physics; they must be re-evaluated with the corrected harness before any scientific claim is attached to them, and no historical record was rewritten. Signed-off-by: mliport agent <noreply@anthropic.com>
This was referenced Nov 6, 2026
hydrogen1222
deleted the
fix/aud-008-harness-identity-and-tier-forwarding
branch
November 7, 2026 00:33
hydrogen1222
pushed a commit
that referenced
this pull request
Nov 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tracking: #61
Audit: #75 (H-01, H-02)
Why
Two defects recorded in the #75 audit ledger were fixed. Neither changes a
scientific threshold or rewrites a historical record.
H-01 — false
failon every GRACE saddle recordvalidation/science/scripts/neb_suite.py::model_identity_checkcomparedmodel_sha256(mliport.model_record.hash_model_path:the file's bytes, or for a directory a tree hash over
sha256(len(rel)||rel||bytes)of every file), withmodel_sha256, which for a directory artifact is thedistributed archive identity (
secondary_hashesverify the individualfiles separately).
For a single-file model (MACE
.model, DPA.pt) those coincide. For a GRACESavedModel directory they can never coincide, so
metrics.model_identity.okwasfalseand the record was written asfaileven when the sampled barrier, CI convergence, active/band force thresholds and
the persistent negative curvature all passed.
Evidence (existing, unmodified record
validation-work/grace/t5/t5_saddle__t5h_saddle_hessian__grace.json):H-02 —
--tiers t3could never runrun_suite.pystartedomat_eval.pywithout its required--subset, so t3died inside the child's argparse.
What changed
common.artifact_sha256— hashes a model artifact with exactly mliport'spath-hashing algorithm, with a regression test proving it agrees with
mliport.model_record.hash_model_pathfor both a file and a directory.EngineContextgained an optionalmodel_pathfield and anartifact_sha256property that fails closed (raises) when the context hasno model path instead of silently falling back to the manifest identity —
that fallback is exactly the H-01 bug.
model_identity_checknow compares like with like, and additionally recordsmanifest_model_sha256andartifact_sha256for provenance.run_suite.pygained--subset(forwarded only toomat_eval.py) andrejects
--tiers t3without it with an actionable message.Tests (red → green)
tests/mliport/mliport/test_science_neb_suite.py, +6 tests:test_artifact_sha256_matches_mliport_path_hashingmliport.model_record.hash_model_pathfor a file and a directorytest_artifact_sha256_is_order_independent_and_content_sensitivetest_model_identity_check_compares_like_with_like_for_directory_artifactsb…, artifacta…, enginea…→ok is Truetest_model_identity_check_still_rejects_a_genuinely_different_artifactc…vs artifacta…→ok is Falsetest_run_suite_forwards_subset_to_omat_eval--tiers t3 --subset Pproducesomat_eval.py … --subset Ptest_run_suite_rejects_t3_without_subset--subsetThe three existing saddle-workflow tests now build a real toy artifact so the
identity check runs end to end (they previously passed a fake
"a"*64).Results
DATA IMPACT
No stored result changed. Historical GRACE saddle records written as
failbecause of this comparison are misclassified, not wrong physics: they must
be re-evaluated with the corrected harness before any scientific claim is
attached, and the old records stay untouched. No threshold was relaxed.
Not done here
AUD-007 (#82/#83) remains an open, owner-decision blocker: config-search
checkpoint restore still replaces non-representative member geometries, and NEB
geometry-level resume is still not full optimiser-state resume. Both PRs are
still Draft and are not touched by this change.