Skip to content

studyrun: power gate, planted truth, autochecks, forecasts and spec/2 - #436

Merged
HarperZ9 merged 1 commit into
mainfrom
feat/studyrun-v2
Nov 4, 2026
Merged

HarperZ9 merged 1 commit into
mainfrom
feat/studyrun-v2

Conversation

@HarperZ9

@HarperZ9 HarperZ9 commented Nov 2, 2026 •

Copy link
Copy Markdown
Owner

studyrun now refuses five more ways a study can look sound and not be: an unreadable verdict path, a gate too weak to bind, a planted truth with the wrong sign, a limit or constant changed after the fact, and forecasts written after the data. Old specs and locks run and verify as before.

What it adds

  • Dotted keys in verdict and ledger paths: escape as a\.b or pass a list path. An unresolved path names the dotted key and stays UNTESTABLE.
  • Power gate (power.py): the lock runs the locked verdict rules on simulated runs and records each verdict's pass rate with a Wilson interval and a binding flag. A path that resolves in no simulated run refuses the lock.
  • Planted truth (planted.py): fits sign and relative size of lagged terms on a calibration panel, or takes a cited fit, and refuses a truth world that plants a sign against it or drops a measured term.
  • Autochecks (autocheck.py, studyrun audit, exit 4): limit lines reread against their cited spans, constants derived by non-study code, post hoc files sealed by hash with their checkpoint.
  • Forecast scoring (forecast.py): Brier and log score per verdict, interval coverage and miss factor per quantity, into result.json and a ledger line.
  • studyrun.spec/2: the same shape as /1, with forecasts for every verdict, power, planted, limits and constants required at lock.
  • ext_sha256 in new locks freezes power, planted and forecasts beside plan_sha256.

docs/STUDYRUN.md documents each one.

Where it came from, and how it was merged

The upgrade was made on 2026-11-02 in the private prototype the Flywheel copy came from, with a carry-over list of files and sha256 prefixes. Every source file matched that list before the move.

  • autocheck.py, forecast.py, planted.py, power.py and verdict.py land byte-identical to the list.
  • __init__, __main__, ledger, prereg, spec and stage had diverged in Canonical-record spec, shared stats vectors and studyrun #433 (canonical-record/1 hashing, the lock's canonical_form, ledger scope keys), so the list says merge by hand. I three-way merged each against the upgrade's pre-change backup: 7 conflicts: four import blocks, the CLI usage text, the lock line and plan_sha256. Flywheel's behaviour is kept in every conflict.
  • One change beyond the upgrade: ext_sha256(spec, form) hashes under the lock's own canonical_form, as plan_sha256 already does, so a lock written by the prototype code verifies here.
  • docs/provenance/studyrun-upgrade-2026-11-02.json lists every file's source and both sha256 values.

Tests

  • tests/test_studyrun_upgrade.py: the upgrade's 23 tests, imports changed to harness.studyrun and tests.test_studyrun. They include a sign-flipped plant that must be refused, an underpowered verdict that must be non-binding, an unescaped dotted key that must stay UNTESTABLE, a removed limit line, a changed constant, an edited post hoc file, an unregistered post hoc file, exact Brier and log scores, spec/2 refusal and the CLI.
  • Studyrun slice plus canonical vectors, local Windows, Python 3.12: 102 passed.
  • Compatibility sweep rerun on the placed copy: the upgrade's own compat_sweep.py, through a shim that imports harness.studyrun, over all 570 studyrun.spec/1 specs in the private workspace, once on Flywheel main and once on this branch. Check problems, plan hashes and every re-judged verdict are identical for all 570; 551 have a stored scored result. The same 3 specs error under both, as the upgrade notes record (two compare a string with a number, one uses ==).
  • Gates run locally: scope migration, data location, file gate, path length, claim language, writing on docs/STUDYRUN.md. CI is the check of record.

Follow-up

power.py brings one more Wilson interval. It should test against tests/data/stats_vectors.json like the Holm and bounds code; not done here.

Not merged; left for review.

🤖 Generated with Claude Code

Adds studyrun.spec/2, verdict paths for keys that hold a dot, a power
gate, planted-truth fitting, autochecks for limits, constants and post
hoc files (studyrun audit, exit 4), forecast scoring and ext_sha256 in
new locks.

The four new modules and verdict.py land byte-identical to the upgrade's
carry-over list. __init__, __main__, ledger, prereg, spec and stage were
three-way merged against the pre-upgrade backup, because the Flywheel
copy had diverged: canonical-record/1 hashing, the lock's
canonical_form, and ledger scope keys stay. ext_sha256 is hashed under
the lock's own canonical form, so a lock written by the prototype code
still verifies.

tests/test_studyrun_upgrade.py carries the upgrade's 23 tests with
Flywheel imports. A compatibility sweep over 570 studyrun.spec/1 specs
gives identical check problems, plan hashes and re-judged verdicts on
Flywheel main and on this branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@HarperZ9
HarperZ9 merged commit 6a2136c into main Nov 4, 2026
79 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant