Repository navigation
test(neb,confsearch): AUD-007 — pin what checkpoint resume really preserves - #82
Conversation
AUD-007 (tracker #75). No production behaviour changes; test-only plus one documentation sentence. confsearch: the engine already checkpoints the per-replica md_rng/mc_rng bit-generator states after every unit and restores them in _restore(). The new test kills a run after every durable unit boundary (1-3 units for one replica / three cycles, 1-4 for two replicas / two cycles) and asserts the resumed run's search_results.json is identical to an uninterrupted run's, apart from model.model_path (not identity). Verified non-vacuous: skipping the RNG restore or perturbing the restored geometry each make it fail. NEB: the checkpoint stores the whole band plus band_sha256, and the loader verifies the checksum and re-validates the band. The new tests show the resumed attempt starts from a byte-identical band, that the interrupted attempt's checkpoints and exports survive, and that the resumed run converges to the same climbing image and saddle. NEB has no stochastic optimizer, so there is no generator state to restore. The checkpoint is geometry_resume_only, so a resume restarts the optimiser instead of replaying its state: in the fixture the barrier is 0.9592721 eV uninterrupted vs 0.9590986 eV resumed (~2e-4 eV). That limitation was already recorded in the checkpoint and run_context.json; document it in docs/workflows/neb.md and its Chinese twin as well. DATA IMPACT: no scientific record invalidated (test-only + docs). Tracking: #75
8afa766 to
d1a8b24
Compare
hydrogen1222
left a comment
There was a problem hiding this comment.
AUD-007 independent review — changes requested before merge (documentation correctness).
The new CPU tests are useful, and the branch CI passed. However, both docs/workflows/neb.md and .zh.md state that geometry-only resume "therefore converges to the same saddle". That is not guaranteed for general NEB landscapes: restarting FIRE without its optimizer state may converge to a different stationary band or may not converge within the remaining budget. The tests show one synthetic MEP converges to a nearby saddle (~1e-3 eV), not a universal guarantee.
Please revise EN/ZH to: "Restores the checkpointed band but restarts optimization; convergence, final saddle and barrier can differ; verify convergence and compare results before interpreting." Update PR wording accordingly. Also this branch is 5 commits behind current main 3d6ca2e; rebase on current main and rerun CI before merging. Remember selected result-JSON equality does not imply all extxyz minima are bitwise identical (see #83). No source algorithm changes needed.
…esume DATA IMPACT: no scientific record invalidated.
DATA IMPACT: no scientific record invalidated.
Preserve existing test coverage and correct geometry-only resume caveat. DATA IMPACT: no scientific record invalidated.
Revalidate against merged #83 confsearch checkpoint schema v2 and subsequent changes. Preserve the independent NEB resume-equivalence regression tests and bilingual caveat. Test-only; scientific outputs remain untouched.
|
Responding to the changes-requested review on this PR (2026-11-05). 1. Documentation over-claim — fixed. 2. Rebased/merged onto current main and CI re-run — done. The branch now 3. Acceptance evidence (independent run, not part of this PR's assertions). AC1 — confsearch resume equivalence. Interrupted at every durable unit AC2 — NEB resume correctness. The resumed attempt's first checkpoint is AC3 — regression. Full suite on this head: 2077 passed, 48 skipped, 0 failed 4. On your #83 caveat. Agreed: the committed tests assert equality of the Residual scope is unchanged: synthetic CPU calculators only; no real-backend |
Tracking: #75 (Living code audit & exhaustive scientific test campaign)
AUD-007 — does checkpoint resume really keep structure, RNG and model identity?
Test-only. No production behaviour changed (one docs clarification, see below).
What was checked
md_rng/mc_rngbit-generator statesstep_*checkpoints)incomplete_attempt_N/)confsearch — exact equivalence
The engine checkpoints after every unit (
checkpoints/confsearch.json) and storesreplica.md_rng.bit_generator.state/replica.mc_rng.bit_generator.state, thenrestores them in
_restore()before the resumed units run.test_resume_equivalence.pykills a run after every durable unit boundary(1–3 units for 1 replica / 3 cycles, and 1–4 units for 2 replicas / 2 cycles) and
asserts the resumed run's
search_results.jsonis identical to anuninterrupted run's — ranking, minima, diagnostics and all — with the single
exception of
model.model_path(where a model is stored is not identity).Mutation checks (each mutation makes the equivalence tests fail, so they are not
vacuous):
bit_generator.staterestore → 2 failures;replica.currentby 0.01 Å → 2 failures.NEB — exact structure, geometry-only path
NEBCheckpointStorewrites the whole band as an ASE trajectory plus aband_sha256, and_read_checkpoint_directoryverifies the checksum before_validate_loaded_bandre-checks atom count, indices, masses, PBC, cell andsymbols. The tests confirm:
interrupted attempt had checkpointed (structure restored verbatim);
step_*checkpoints and its exports survive the resume;status=completed,converged=True) to the sameclimbing image and the same saddle.
NEB has no stochastic optimizer; the only randomness is the gradient pre-flight,
seeded from a module constant, so there is no generator state to checkpoint.
Known, documented limitation (not a bug): the checkpoint declares
geometry_resume_only: True, so a resume restarts the NEB optimisation fromthe checkpointed geometry instead of replaying the optimiser state. In the test
fixture the barrier is
0.9592721 eVuninterrupted vs0.9590986 eVresumed —a ~2e-4 eV difference. The test therefore asserts the same saddle within a loose
1e-3 eVguard (explicitly not a precision claim). This was already recordedin the checkpoint and
run_context.json; this PR adds the missing sentence todocs/workflows/neb.md/.zh.mdso the contract is explicit for users too.Mutation check: perturbing the restored band by 0.001 Å makes
test_resume_restores_the_checkpointed_band_exactlyfail.Evidence
main@c7880a2c(rebased after fix(analysis): refuse non-integral md.csv step values instead of truncating (AUD-001) #76/fix(xdatcar): preserve PBC provenance and block unverified bulk analysis (AUD-002) #77/fix(opt): report a crashed optimization as failed, not non-converged (AUD-003) #78/fix(batch): preserve failed and non-converged optimization outcomes #80 merged), Python 3.12.13,pytest 9.1.1, CPU-only
.venv-dev.pytest tests/mliport/mliport/neb/test_neb_resume_equivalence.py→ 4 passed.pytest tests/mliport/mliport/confsearch/test_resume_equivalence.py→ 3 passed.test_resume_identity.py→ 23 passed.ruff check/format --check,generate_docs.py --check,check_repository_hygiene.pyandtest_docs_contract.pyall clean.DATA IMPACT
No scientific record invalidated (test-only + docs). No stored trajectory,
checkpoint or evidence file is touched.
Residual
test_resume_converges_to_the_same_saddleuses a synthetic curved MEP; itverifies the resume contract, not model accuracy.