fix(evals): enforce paid blinding boundary

Author: ernestprovo23

docs/plans/2026-07-17-h1-method-hardening-design.md

@@ -1,7 +1,7 @@
 # H1 Method Hardening Design
 
-**Linear:** DSE-708  
-**Dependency:** PR #52 at `4365c28`  
+**Linear:** DSE-708
+**Dependency:** PR #52 at `4365c28`
 **Gate:** Required before any paid study call
 
 ## Decision
@@ -19,17 +19,17 @@ Confirmatory reports fail closed if any frozen field is missing or its observed
 
 ## Experimental unit and randomization
 
-The unit of analysis is the task. Roster and replicate are repeated measurements averaged within task. Create every task x roster x condition x replicate cell before execution. Derive an independent condition permutation from the master seed plus task and roster IDs; interleave blocks so time or provider order cannot align with one condition. All failed, timed-out, malformed, over-budget, abstained, incomplete, ungraded, and unresolved cells remain failures.
+The unit of analysis is the task. Roster and replicate are repeated measurements averaged within task. Create every task x roster x condition x replicate cell before execution. Derive an independent condition permutation from the master seed plus task and roster IDs so condition order varies across blocks. All failed, timed-out, malformed, over-budget, abstained, incomplete, ungraded, and unresolved cells remain failures.
 
 ## Grading contract
 
 Two independent graders score every successful output. Atomic records include rubric item, error category and severity, severe-error flag, holistic dimensions, reviewer seconds, confidence or abstention, rubric version/hash, grader batch/order, and condition/provider guess. Adjudication sees only genuine disagreements, cites every source record, and never overwrites raw judgments.
 
-Non-success records bypass the human queue and are automatic failures. Successful grader views strip provider, model, condition, mode, and formatting fingerprints; normalize presentation deterministically; run a leakage scan; and retain a separate hashed/access-controlled blind map.
+Non-success records bypass the human queue and are automatic failures. Successful grader views expose only normalized presentation text, scan it for predeclared provider/model/condition labels, and retain a separate hashed/access-controlled blind map. Paid and confirmatory scoring rejects direct planned-run targets, unhashed maps, and maps that do not cover exactly the successful output set.
 
 ## Analysis and gates
 
-Study reporting uses failure-inclusive Wilson rates, task-clustered paired bootstrap differences with roster/replicate averaging, raw agreement, prevalence, kappa, adjudication rate, severe errors, reviewer effort, token/cost distributions, latency distributions, deviations, and leakage guesses.
+Study reporting uses failure-inclusive Wilson rates, task-clustered paired bootstrap differences with roster/replicate averaging, raw agreement, prevalence, kappa, adjudication rate, severe errors, reviewer effort, total and analysis-set cost, p95 latency, deviations, and retained leakage guesses. A paid workflow must preregister and apply its material-leakage rule before claiming method readiness.
 
 Reliability is undefined—not perfect—when expected agreement is one. A future paid pilot is
 method-ready only with at least 95% double grading, raw agreement at least 80%, kappa at least

docs/plans/2026-07-17-h1-method-hardening.md

@@ -16,7 +16,7 @@
 
 ## Phase 3 — Confirmatory estimands and refusal gates
 
-1. Add tests for task-level roster averaging, task-clustered paired bootstrap, one-sided severe-error and effort/latency noninferiority boundaries, symmetric task exclusions, deviations, cost/latency distributions, and report refusal on freeze drift or exploratory evidence.
+1. Add tests for task-level roster averaging, task-clustered paired bootstrap, one-sided severe-error and effort/latency noninferiority boundaries, symmetric task exclusions, deviations, complete cost/latency receipts and summaries, and report refusal on freeze drift or exploratory evidence.
 2. Extend `scoring.py`, `reporting.py`, and CLI artifact validation. Archive every gate input and seed.
 3. Re-run focused tests and commit.
 

src/conclave/evals/scoring.py

@@ -912,7 +912,19 @@ def score_study(
     if len({item.adjudication_id for item in adjudications}) != len(adjudications):
         raise ValueError("adjudication_id values must be unique")
 
+    paid_study = manifest.evidence_classification != "synthetic_exploratory"
+    if paid_study:
+        if blind_map is None or blind_map.blind_map_hash is None:
+            raise ValueError("paid scoring requires a hashed blind map")
+        if any(item.planned_run_id is not None for item in (*raw_judgments, *adjudications)):
+            raise ValueError("paid judgments and adjudications require opaque output IDs")
     mapping = _target_map(blind_map)
+    if paid_study:
+        successful_run_ids = {
+            record.planned_run_id for record in study_run.records if record.outcome == "success"
+        }
+        if set(mapping.values()) != successful_run_ids or len(mapping) != len(successful_run_ids):
+            raise ValueError("paid blind map must cover exactly the successful output set")
     judgments_by_run: dict[str, list[GraderJudgment]] = defaultdict(list)
     judgment_ids = {item.judgment_id for item in raw_judgments}
     seen_grader_targets: set[tuple[str, str]] = set()

tests/evals/test_analysis_gates.py

@@ -14,7 +14,7 @@
     recompute_simultaneous_upper_bounds,
     score_study,
 )
-from tests.evals.test_method_grading import _judgment
+from tests.evals.test_method_grading import _blind_targets, _judgment
 from tests.evals.test_study_design import _design, _tasks
 
 
@@ -74,16 +74,22 @@ def _confirmatory_report(*, minimum_tasks: int = 2):
         frozen_design=design,
     )
     study_run = _confirmatory_report_run(manifest)
+    blind_map, opaque_by_run = _blind_targets(study_run)
     records = study_run.records
     judgments = tuple(
-        _judgment(record.planned_run_id, grader, value=planned.condition_id != "self_refine")
+        _judgment(
+            opaque_by_run[record.planned_run_id],
+            grader,
+            value=planned.condition_id != "self_refine",
+        )
         for planned, record in zip(manifest.planned_runs, records, strict=True)
         for grader in ("grader-a", "grader-b")
     )
     report = score_study(
         manifest=manifest,
         study_run=study_run,
         raw_judgments=judgments,
+        blind_map=blind_map,
         bootstrap_seed=design.bootstrap.seed,
         bootstrap_samples=design.bootstrap.samples,
     )
@@ -115,11 +121,14 @@ def _confirmatory_report_run(manifest):
 
 
 def _gate(manifest, report):
+    study_run = _confirmatory_report_run(manifest)
+    blind_map, _ = _blind_targets(study_run)
     return evaluate_confirmatory_gates(
         manifest=manifest,
-        study_run=_confirmatory_report_run(manifest),
+        study_run=study_run,
         raw_judgments=report.raw_judgments,
         report=report,
+        blind_map=blind_map,
     )
 
 
@@ -253,8 +262,11 @@ def test_confirmatory_gate_recomputes_secondary_bounds_from_task_inputs(
 
 def test_caller_cannot_substitute_optimistic_bounds_to_turn_redesign_into_go() -> None:
     manifest, report = _confirmatory_report()
-    condition_by_run = {
-        planned.planned_run_id: planned.condition_id for planned in manifest.planned_runs
+    study_run = _confirmatory_report_run(manifest)
+    blind_map, opaque_by_run = _blind_targets(study_run)
+    condition_by_output = {
+        opaque_by_run[planned.planned_run_id]: planned.condition_id
+        for planned in manifest.planned_runs
     }
     severe = AtomicError(
         rubric_item="safety",
@@ -270,15 +282,15 @@ def test_caller_cannot_substitute_optimistic_bounds_to_turn_redesign_into_go() -
                 "severe_error": True,
             }
         )
-        if condition_by_run[judgment.planned_run_id] == "elite_full"
+        if condition_by_output[judgment.opaque_output_id] == "elite_full"
         else judgment
         for judgment in report.raw_judgments
     )
-    study_run = _confirmatory_report_run(manifest)
     redesign_report = score_study(
         manifest=manifest,
         study_run=study_run,
         raw_judgments=bad_judgments,
+        blind_map=blind_map,
         bootstrap_seed=manifest.frozen_design.bootstrap.seed,
         bootstrap_samples=manifest.frozen_design.bootstrap.samples,
     )
@@ -288,6 +300,7 @@ def test_caller_cannot_substitute_optimistic_bounds_to_turn_redesign_into_go() -
             study_run=study_run,
             raw_judgments=bad_judgments,
             report=redesign_report,
+            blind_map=blind_map,
         ).status
         == "REDESIGN"
     )
@@ -322,6 +335,7 @@ def test_caller_cannot_substitute_optimistic_bounds_to_turn_redesign_into_go() -
             study_run=study_run,
             raw_judgments=bad_judgments,
             report=substituted,
+            blind_map=blind_map,
         )
 
 
@@ -405,6 +419,8 @@ def test_confirmatory_gate_uses_single_model_latency_baseline_and_reliability()
 
 def test_rotating_grader_pairs_still_produce_complete_reliability() -> None:
     manifest, report = _confirmatory_report()
+    study_run = _confirmatory_report_run(manifest)
+    blind_map, _ = _blind_targets(study_run)
     raw = []
     for index, judgment in enumerate(report.raw_judgments):
         pair = index // 2
@@ -418,8 +434,9 @@ def test_rotating_grader_pairs_still_produce_complete_reliability() -> None:
         )
     rescored = score_study(
         manifest=manifest,
-        study_run=_confirmatory_report_run(manifest),
+        study_run=study_run,
         raw_judgments=tuple(raw),
+        blind_map=blind_map,
         bootstrap_seed=manifest.frozen_design.bootstrap.seed,
         bootstrap_samples=manifest.frozen_design.bootstrap.samples,
     )