Add run --prove-fail: execute every check against a declared known-bad state - #104
Open
heyjawrsh wants to merge 2 commits into
Open
Add run --prove-fail: execute every check against a declared known-bad state#104heyjawrsh wants to merge 2 commits into
heyjawrsh wants to merge 2 commits into
Conversation
…d state --baseline proves a check does not false-FAIL on an unmodified tree. Nothing proved the mirror: that a check does not false-PASS on a broken one. A check that cannot fail is a task that cannot be verified. Each task may declare known_bad, a shell command that mutates its scratch dir into the failure the check claims to detect. --prove-fail runs it, then runs the task's real check against that state, in a fresh scratch dir (a detached worktree when the manifest uses worktrees). The check FAILING is the good outcome, reported as proved with the failure output shown -- that output is what a retry prompt would receive. A check that PASSES on known-bad work is reported BROKEN. Unlike --baseline, which always exits 0 and leaves judgment to the orchestrator, exit is nonzero when anything is broken, inconclusive or errored: a pass on a known-bad state is not a judgment call. The two flags answer different questions and refuse to run together. Spawns no workers and writes no eval rows.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A check of mine passed against a state I had deliberately broken — it grepped a README that already contained the words from earlier work, so it could not tell new material from old. --baseline had nothing to say about it: baseline proves a check does not false-FAIL on an unmodified tree, and nothing proved the mirror.
--prove-fail is that mirror. Each task may declare known_bad, a shell command that mutates its scratch dir into the failure the check claims to detect. The mode runs it, then runs the task's real check against that state, in a fresh scratch dir (a detached worktree when the manifest uses worktrees). The check failing is the good outcome — reported as proved, with the failure output shown, since that output is what a retry prompt receives. A check that passes on known-bad work is reported BROKEN.
Unlike --baseline, exit is nonzero when anything is broken, inconclusive, or errored: a pass on a known-bad state is not a judgment call to defer. The two flags refuse to run together — they answer different questions. Spawns no workers, writes no eval rows.
Tests: tests/test_prove_fail_mode.py, seven hermetic cases (proved, broken, skipped, failing known_bad, inconclusive, field validation, flag conflict), modeled on test_baseline_mode.py —nonexistent engine binary so nothing can spawn, RINGER_NO_SELF_UPDATE=1. Full suite green on macOS. Rebased on current main.