Skip to content

fix(confsearch): fail closed on a damaged checkpoint and keep the minima mask (AUD-007) - #83

Merged
hydrogen1222 merged 13 commits into
mainfrom
fix/aud-007-confsearch-resume-integrity
Nov 7, 2026
Merged

hydrogen1222 merged 13 commits into
mainfrom
fix/aud-007-confsearch-resume-integrity

Conversation

@hydrogen1222

@hydrogen1222 hydrogen1222 commented Nov 2, 2026 •

Copy link
Copy Markdown
Owner

Tracking: #61 / #75 — AUD-007.

What this PR now fixes

  1. Damaged confsearch checkpoints are refused (out-of-range unit index, missing RNG streams, malformed JSON/schema) instead of silently producing truncated or divergent searches.
  2. The fixed-atom mask is restored on every exported minimum.
  3. Confirmed scientific-data integrity defect fixed: each registry member now serializes its own atom numbers, positions, cell, PBC and full classification. On resume, that member is reconstructed with its own coordinates rather than borrowing the representative's coordinates while retaining its own energy/origin. Input checks fail closed on missing/inconsistent/nonfinite member geometry.
  4. Checkpoint layout is upgraded to schema/revision 2. Existing schema-1 checkpoints are not resumable: they have already lost per-member geometries. Historical files remain untouched for forensic review. The selected output and full minima/all_minima.extxyz are now compared against an uninterrupted synthetic-CPU reference; the previous strict xfail has been removed because this defect is repaired.

The original checkpoint corruption/mask checks and one independent regression for old schema refusal remain. This PR now targets the latest main including NEP #88/#89 and harness #87.

DATA IMPACT

Historical resumed confsearch output may be scientifically wrong. In the earlier synthetic reproduction, 3 of 21 members carried representative positions instead of their original geometries (maximum 0.19 Å difference) while retaining per-member energies. That existing issue is now fixed for new schema-2 checkpoints. Previously exported member geometries cannot be recovered from old schema-1 checkpoints alone; inspect the original quench raw files if available. Do not relabel prior runs as corrected. Existing schema-1 checkpoint resumability is intentionally refused rather than fabricating lost data.

Validation boundary

CI covers synthetic CPU fixtures only. Real-model checkpoint/resume and GPU trajectories require separate hardware verification before marking AUD-007 fully complete.

…ima mask

AUD-007 (tracker #75).  Three defects found by CPU fault injection against the
synthetic egg-crate search; no integrator, model, energy or force semantics
change.

1. A damaged checkpoint was resumed instead of refused.  A unit index beyond
   the plan made the unit loop skip everything and report a *truncated* search
   as ``completed`` (21 quenches/15 minima in the reference vs 15/12 with
   unit_index=10**6); an empty or partial ``rng_states`` silently re-seeded the
   replicas and produced a different ranking, and a missing stream raised a raw
   KeyError.  The engine now validates the unit index against the plan, the
   replica set and the per-replica md/mc generator states before restoring, and
   refuses with ConfsearchPreparationError, leaving the checkpoint untouched.

2. A truncated checkpoint file raised a bare JSONDecodeError, and the schema
   was never checked.  ConfsearchCheckpointStore.load() now raises a domain
   error for unreadable JSON, non-object payloads and an unexpected schema.

3. minima/all_minima.extxyz was not reproducible across a resume.  A
   checkpoint round trip rebuilds registry geometries from bare
   numbers/positions/cell/PBC, so the fixed-atom mask survived only on the
   minima registered after the resume; the file is now written from the run's
   own regions, which is deterministic for every minimum.

DATA IMPACT: no scientific record invalidated.  Energy, force, geometry and
ranking values are unchanged; only a damaged checkpoint is refused instead of
reinterpreted, and the extxyz move_mask of a resumed run is corrected to match
an uninterrupted one.

Tracking: #75

@hydrogen1222 hydrogen1222 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AUD-007 independent review — valuable partial fix; outstanding scientific-data issue must stay explicit.

I verified source: UniqueMinimum.to_dict() writes positions only for the representative, while MinimumRegistry.from_dict() reconstructs every member with atoms=geometry.copy(). This means an interrupted/resumed all_minima.extxyz may associate a member's recorded energy/origin with the representative's (different) geometry. Your strict xfail (3/21 frames, up to 0.19 Å in the CPU fixture) is a confirmed scientific-record integrity issue, NOT a passing test.

The fail-closed checkpoint validation and fixed-mask repair are useful and have 11/11 green GitHub checks, but the PR's DATA IMPACT section must not claim "geometry values unchanged / historical runs unaffected." Qualify that preexisting resume-generated member geometries can already be inaccurate and need provenance assessment. test_resume_reproduces_the_reference_artifacts_bitwise presently compares only search_results.json, registry.json and discovery.csv, not all_minima.extxyz; please describe its scope accurately.

Recommendation: do NOT close AUD-007 as done. Log the member-geometry defect as a separate high-priority follow-up with a decision on CS_CHECKPOINT_SCHEMA_REVISION and old-checkpoint compatibility. Because this PR does not introduce that defect, the targeted fixes can be evaluated for incremental merge after correcting DATA IMPACT and explicitly tracking the failing regression. Owner approval required before a schema-bumping fix; do not silently fabricate member geometries from legacy checkpoints.

…assification

DATA IMPACT: historical resumed runs may have mismatched member energies/coordinates and must be reviewed; original files preserved.
…patible old resumes

DATA IMPACT: historical checkpoints remain readable as raw evidence but cannot resume under v2.
…patible old resumes

DATA IMPACT: historical checkpoints remain readable as raw evidence but cannot resume under v2.
…rupted export

DATA IMPACT: old schema-1 checkpoints intentionally blocked on resume; historical artifacts remain untouched.
…r with latest main

DATA IMPACT: schema-1 checkpoints cannot safely resume; historical post-resume member geometries may need review.
DATA IMPACT: no scientific record invalidated.
…eometry v2

DATA IMPACT: schema-1 historical registries remain unchanged.
…eometry v2

DATA IMPACT: schema-1 historical registries remain unchanged.
…eometry v2

DATA IMPACT: schema-1 historical registries remain unchanged.
DATA IMPACT: no historical artifact rewritten.
@hydrogen1222
hydrogen1222 marked this pull request as ready for review November 7, 2026 04:59
@hydrogen1222
hydrogen1222 merged commit 2d0cbea into main Nov 7, 2026
12 checks passed
@hydrogen1222
hydrogen1222 deleted the fix/aud-007-confsearch-resume-integrity branch November 7, 2026 04:59
hydrogen1222 added a commit that referenced this pull request Nov 7, 2026
…nflict

Keep both #83 checkpoint integrity notes and #90 change notes, retaining latest main code and isolated feature files.

DATA IMPACT: no scientific record rewritten.
hydrogen1222 added a commit that referenced this pull request Nov 7, 2026
…ngelog conflict

Preserve #83 data-integrity documentation and #91 trusted conversion notes with new DPA files.

DATA IMPACT: none.
hydrogen1222 added a commit that referenced this pull request Nov 7, 2026
Resolve changelog conflict while retaining NEP89 #90, AUD-007 #83, and DPA #91 notes; keep latest main files except the four DPA-specific additions.

DATA IMPACT: no scientific record rewritten.
hydrogen1222 added a commit that referenced this pull request Nov 14, 2026
Revalidate against merged #83 confsearch checkpoint schema v2 and subsequent changes. Preserve the independent NEB resume-equivalence regression tests and bilingual caveat.

Test-only; scientific outputs remain untouched.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant