v4.1 guidance baseline + Phase 5–10 eval evidence chain#17
Merged
Conversation
14-phase flow for a 100K window: P1-P6 reproduce the standard arc below the 30% nudge threshold (regime A, nudge silent), P7-P11 climb the pressure gradient (regime B), P12-P14 observe whether ACM re-fires after a pressure-forced travel (regime C, habit vs reflex). eval/regimes.mjs assigns regime boundaries mechanically from events.jsonl token readings and the first successful acm_travel.
…es 3/4, kimi immune Five-model p5gradient round at a 100K window (n=1). opus folded at 21% with zero nudges (pure 道) and peaked at 36%; sol/mimo/deepseek folded nudge-responsively; kimi ignored all three tiers to 83%. Contract findings: shallow folds + same-turn regrowth silently disarm the nudge cycle; fold depth is the hidden variable; compaction reachable at 96% for verbose models. Also fixes regimes.mjs: max-per-turn context (mid-turn travels hid tier crossings) and post-travel regime labeling when travel precedes the threshold crossing.
…irst sample A successful travel reset the reminder cycle and established the new baseline from the first real post-transition usage. A shallow fold followed by same-turn regrowth could land that first sample in a higher tier, which the baseline then recorded as highestReachedLevel — silently consuming reminders the cycle never sent (observed in the p5gradient eval: mimo folded 53.4K→44.8K, regrew to 52.7K in the same turn, and the 30%/50% tiers went silent while context climbed back to 56%). The travel tool now seeds the pending cycle's baseline tier from its own verified landing estimate. The first real usage still establishes and persists the baseline (real tokens, seeded level), but the seeded tier — not the sample's — becomes highestReachedLevel, keeping tiers above the landing point armed. Unseeded cycles (compaction, manual /tree) keep the sampled-level behavior.
…0/6 vs baseline 2/2) V-A cruise clarity, V-B fold depth, V-C chain-start: zero folds in 6 runs. Baseline (375fe52, same guidance + baseline-seeding fix) folded 2/2 — one clean nudge-responsive mid-depth fold, one over-fold that lost the P5 answer (fold-honesty guard doesn't reach kimi). Fix validated: post-travel nudge cycle re-arms correctly. kimi first-fold is high-variance and not text-addressable; remaining lever is an active non-pressure signal.
…s 道, opus-4.6 pressure-untested gpt-5.5 folds below the nudge threshold in BOTH windows (purest 道-driven folder); terra folds sub-threshold then rebases to root (3/3); luna folds nudge-adjacent; glm/mimo/dspro save-at-most; opus-4.6 zero ACM but its 200K run never reached the 30% tier. Fold/nudge ordering verified mechanically from events.jsonl. Task completion 3/3 across all 14 runs.
…context swapped buildFullEnvAgentDir replicates the user's real agent dir (16 packages, extensions, skills, global AGENTS.md) minus the installed pi-context, with heavy stores symlinked and the model allowlist removed; the driver drops the --no-* flags in full-env mode; the workspace moves to a tmpdir so cwd-ancestry discovery cannot leak the repo's own AGENTS.md (the ACM design doc) into the tested agent. mcp.json excluded. Smoke-validated: startup, model routing, and turn completion all healthy.
…ripts run-flow.mjs only judges at run time; when the judge reply fails to parse, the transcript is on disk but there was no way to retry without re-running the whole flow. rejudge.mjs rebuilds the exact judge prompt from transcript.txt + the flow definition and updates verdict.json/report.json in place. Exports buildJudgePrompt from judge.mjs for reuse.
head (v3) wins: only column where all four internalizers fold (2 pure-dao, zero pathology). fable second (3/4 folds, all nudge-late). pr12 high-variance (one best cell, one overfold thrash, one spam). pr14 suppresses the fold for all four models (0/4). fableprev is the pathology column (thrash or spam in every cell).
4 replication runs (opus-head x2, terra-head x2) before any v4 text work: terra is a stable nudge-responsive folder (3/3 samples: fold P8-P10, handoff 3/3, 2/3 strong). opus is bimodal: 2/3 samples do the identical P10 nudge-responsive fold, 1/3 goes fully zero-ACM substituting ~/.pi/backups file backups. All 3 opus samples show the file-backup channel competing with acm_checkpoint — backup-channel disambiguation becomes the top v4 text change.
Three additive changes, every pre-existing doctrine sentence untouched: 1. Backup-channel disambiguation in Save: 'File backups protect the disk; a checkpoint protects this conversation.' Phase-8 replication showed all 3 opus-head samples substituting ~/.pi/backups file backups for acm_checkpoint (reco<=1; one run fully zero-ACM). 2. Fold self-check: 'Before traveling, answer in one line: what leaves the working set, what pointer recovers it, and what single action is NEXT.' Borrows pr12's fold criteria as a judgment aid, not a gate. 3. Two seam cues: final answer due -> rebase check; archived detail needed -> rehydrate. terra-head missed both moments in all 3 samples. No mandatory preflight, no transition tables, no suffix naming — the machinery that made pr12/fableprev thrash stays banned. Length guard raised 6000 -> 6500 with the rationale recorded in the test.
v4 (3 additive CORE lines) validated on 10 models + 4 replications: luna first perfect score, glm first fold, terra reco/ceiling to 3 — but terra regression pre-registered as confirmed (2/3 samples task-degraded vs 0/3 under v3). Failure shape: folding while the current turn's obligation is unfulfilled. Mechanism isolated to the fold self-check; seam cues clean. v4.1 proposal: add the missing fourth condition (never fold away an unfulfilled promise).
…ition Phase-9 isolated v4's regression vector: 3/10 internalizer samples folded while the current turn's obligation was unfulfilled (red tests, unanswered question, half-landed change), vs 0/8 under v3. The v4 self-check was satisfiable mid-obligation — NEXT is always writable. Add the missing fourth condition: never fold away an unfulfilled promise to the user.
Phase-9 P13 audit (8 folding runs): 6/8 handoffs carried the P7 semantics in State — answering directly was correct, not a missed rehydrate; one run re-verified empirically (equally valid); zero true misses. The v1 rubric dinged these as ceiling failures. v2: handoff-carried answers are correct, empirical re-verification counts, rehydrate is the fallback for details genuinely absent from the working set. timing also explicitly dings folding while the current turn's obligation is unfulfilled (the v4 regression vector). Dimension scores not comparable with v1-judged runs.
Pre-registered adoption test (terra x3 + sol x2 + luna + glm, rubric v2): 5/5 regression-group cells at task 3/3 (v4 had 3 task-degraded there). Mechanism correction: v4.1 does not prevent bad-moment folds, it prevents folds from losing the promise. sol = cleanest phenotype of the series (2x 3/3, timing 3/3 both). glm's v4 fold unreplicated (bimodal; head baseline also zero). v4.1 >= head on every measured cell.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
14 commits: the complete evidence-driven guidance iteration from Phase-5 through Phase-10, ending in the v4.1 guidance baseline (adopted after pre-registered validation).
Guidance changes (production)
db756413): three additive CORE lines — backup-channel disambiguation (file backups ≠ checkpoints), fold self-check (what leaves / what pointer / what NEXT), two seam cues (final answer → rebase check; archived detail → rehydrate)afc4a6ea): self-check gains the promise-keeping condition — never fold away an unfulfilled promise (red tests, unanswered question, half-landed change)375fe529): seed post-travel baseline from landing estimateEval infrastructure
--full-envharness (c0383ee): production-fidelity env with only pi-context swappedrejudge.mjs: re-judge runs from persisted transcriptsEvidence chain (eval/PHASE5–10 logs)
Pre-push review
Two-axis review (code-review skill), range
origin/main..HEAD: Standards PASS / Spec PASS, zero findings.Verification
bun run verify:acmfull gate green at every guidance commit (110 root tests + 20 guidance tests + typecheck + 28 host-fixture tests). All behavioral claims backed by mechanical facts from run events.jsonl, not judge impressions.