Repository navigation
fix(confsearch): fail closed on a damaged checkpoint and keep the minima mask (AUD-007) - #83
Conversation
…ima mask AUD-007 (tracker #75). Three defects found by CPU fault injection against the synthetic egg-crate search; no integrator, model, energy or force semantics change. 1. A damaged checkpoint was resumed instead of refused. A unit index beyond the plan made the unit loop skip everything and report a *truncated* search as ``completed`` (21 quenches/15 minima in the reference vs 15/12 with unit_index=10**6); an empty or partial ``rng_states`` silently re-seeded the replicas and produced a different ranking, and a missing stream raised a raw KeyError. The engine now validates the unit index against the plan, the replica set and the per-replica md/mc generator states before restoring, and refuses with ConfsearchPreparationError, leaving the checkpoint untouched. 2. A truncated checkpoint file raised a bare JSONDecodeError, and the schema was never checked. ConfsearchCheckpointStore.load() now raises a domain error for unreadable JSON, non-object payloads and an unexpected schema. 3. minima/all_minima.extxyz was not reproducible across a resume. A checkpoint round trip rebuilds registry geometries from bare numbers/positions/cell/PBC, so the fixed-atom mask survived only on the minima registered after the resume; the file is now written from the run's own regions, which is deterministic for every minimum. DATA IMPACT: no scientific record invalidated. Energy, force, geometry and ranking values are unchanged; only a damaged checkpoint is refused instead of reinterpreted, and the extxyz move_mask of a resumed run is corrected to match an uninterrupted one. Tracking: #75
hydrogen1222
left a comment
There was a problem hiding this comment.
AUD-007 independent review — valuable partial fix; outstanding scientific-data issue must stay explicit.
I verified source: UniqueMinimum.to_dict() writes positions only for the representative, while MinimumRegistry.from_dict() reconstructs every member with atoms=geometry.copy(). This means an interrupted/resumed all_minima.extxyz may associate a member's recorded energy/origin with the representative's (different) geometry. Your strict xfail (3/21 frames, up to 0.19 Å in the CPU fixture) is a confirmed scientific-record integrity issue, NOT a passing test.
The fail-closed checkpoint validation and fixed-mask repair are useful and have 11/11 green GitHub checks, but the PR's DATA IMPACT section must not claim "geometry values unchanged / historical runs unaffected." Qualify that preexisting resume-generated member geometries can already be inaccurate and need provenance assessment. test_resume_reproduces_the_reference_artifacts_bitwise presently compares only search_results.json, registry.json and discovery.csv, not all_minima.extxyz; please describe its scope accurately.
Recommendation: do NOT close AUD-007 as done. Log the member-geometry defect as a separate high-priority follow-up with a decision on CS_CHECKPOINT_SCHEMA_REVISION and old-checkpoint compatibility. Because this PR does not introduce that defect, the targeted fixes can be evaluated for incremental merge after correcting DATA IMPACT and explicitly tracking the failing regression. Owner approval required before a schema-bumping fix; do not silently fabricate member geometries from legacy checkpoints.
…assification DATA IMPACT: historical resumed runs may have mismatched member energies/coordinates and must be reviewed; original files preserved.
…patible old resumes DATA IMPACT: historical checkpoints remain readable as raw evidence but cannot resume under v2.
…patible old resumes DATA IMPACT: historical checkpoints remain readable as raw evidence but cannot resume under v2.
…rupted export DATA IMPACT: old schema-1 checkpoints intentionally blocked on resume; historical artifacts remain untouched.
…r with latest main DATA IMPACT: schema-1 checkpoints cannot safely resume; historical post-resume member geometries may need review.
DATA IMPACT: none.
DATA IMPACT: no scientific record invalidated.
…flow DATA IMPACT: none.
…eometry v2 DATA IMPACT: schema-1 historical registries remain unchanged.
…eometry v2 DATA IMPACT: schema-1 historical registries remain unchanged.
…eometry v2 DATA IMPACT: schema-1 historical registries remain unchanged.
DATA IMPACT: no historical artifact rewritten.
Revalidate against merged #83 confsearch checkpoint schema v2 and subsequent changes. Preserve the independent NEB resume-equivalence regression tests and bilingual caveat. Test-only; scientific outputs remain untouched.
Tracking: #61 / #75 — AUD-007.
What this PR now fixes
minima/all_minima.extxyzare now compared against an uninterrupted synthetic-CPU reference; the previous strict xfail has been removed because this defect is repaired.The original checkpoint corruption/mask checks and one independent regression for old schema refusal remain. This PR now targets the latest main including NEP #88/#89 and harness #87.
DATA IMPACT
Historical resumed confsearch output may be scientifically wrong. In the earlier synthetic reproduction, 3 of 21 members carried representative positions instead of their original geometries (maximum 0.19 Å difference) while retaining per-member energies. That existing issue is now fixed for new schema-2 checkpoints. Previously exported member geometries cannot be recovered from old schema-1 checkpoints alone; inspect the original quench raw files if available. Do not relabel prior runs as corrected. Existing schema-1 checkpoint resumability is intentionally refused rather than fabricating lost data.
Validation boundary
CI covers synthetic CPU fixtures only. Real-model checkpoint/resume and GPU trajectories require separate hardware verification before marking AUD-007 fully complete.