Build benchmark evaluation infrastructure - #328
Closed
kaoru0822-kitauji wants to merge 4 commits into
Closed
Conversation
morluto
marked this pull request as ready for review
August 2, 2026 06:53
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 10d110a596
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Owner
morluto
added a commit
that referenced
this pull request
Aug 2, 2026
Bump the agent-workflow-v1 dataset version from 1.1.0 to 1.2.0 to satisfy the suite version-bump planner gate introduced by PR #328. The dataset manifest and task digests are unchanged because the version field lives in suite.toml, not in the generated dataset.toml. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
morluto
added a commit
that referenced
this pull request
Aug 2, 2026
Bump the agent-workflow-v1 dataset version from 1.1.0 to 1.2.0 to satisfy the suite version-bump planner gate introduced by PR #328. The dataset manifest and task digests are unchanged because the version field lives in suite.toml, not in the generated dataset.toml. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
morluto
added a commit
that referenced
this pull request
Aug 2, 2026
Bump the agent-workflow-v1 dataset version from 1.1.0 to 1.2.0 to satisfy the suite version-bump planner gate introduced by PR #328. The dataset manifest and task digests are unchanged because the version field lives in suite.toml, not in the generated dataset.toml. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
morluto
added a commit
that referenced
this pull request
Aug 2, 2026
Bump the agent-workflow-v1 dataset version from 1.1.0 to 1.2.0 to satisfy the suite version-bump planner gate introduced by PR #328. The dataset manifest and task digests are unchanged because the version field lives in suite.toml, not in the generated dataset.toml. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 10d110a596
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
morluto
added a commit
that referenced
this pull request
Aug 2, 2026
Enforce an assurance-boundary limitation disclaimer in the indexed-pairwise-vacuity verifier, matching the established limsup-quantifier-alignment and convergence-mode-separation pattern. The verifier now requires the submission limitations list to mention the open conjecture, preventing overclaiming of the finite-model audit as a full settlement of the indexed pairwise vacuity question. Add regression tests that accept the canonical submission (with the "open conjecture" disclaimer) and reject an overclaiming limitations list that omits the disclaimer. Bump the agent-workflow-v1 dataset version from 1.1.0 to 1.2.0 to satisfy the suite version-bump planner gate introduced by PR #328. Regenerate the task digest, Dockerfile verifier checksum label, dataset manifest, and research ledger manifest digests. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
morluto
added a commit
that referenced
this pull request
Aug 2, 2026
Address the eleven unresolved review threads on PR #328: - Enforce exactly three pilot repetitions in the held-out manifest validator so a bumped manifest cannot widen the pilot without re-freezing the experiment (Vu6-P). - Add validate_pilot_gate and require a VALID, pair-complete, non-ceiling pilot comparison report before the decision stage can be rendered, both in the bundle CLI and the heldout-benchmarks workflow (Vu6-U). - Require a positive n_completed_trials stat in the held-out runner and cross-check it against the per-trial payload count so an incomplete Harbor result cannot be promoted to COMPLETE (Vu6-Q). - Reconcile existing complete result directories before rerunning a resumed trial so an interrupted workflow does not silently re-execute finished pairs (VuxoL). - Preserve unrelated docker-compose differences in the comparison signature by stripping only the c2 condition overlay instead of every non-proxy compose file (Vu6-O). - Bind observed agent name, agent version, and model provider identities in both observation and held-out evidence, and reject drift against the frozen experiment or runtime snapshot (Vu6-S). - Reject comparison reports missing any of the five required outcome dimensions (correctness, evidence_validity, scope_accuracy, assurance_calibration, false_certification) (Vu6-V). - Require dataset version bumps before forced full planning so a workflow_dispatch cannot bypass the version-value check (VuxoK). - Pass OPENAI_API_KEY to the held-out execution step so model authentication is available in the protected environment (VuxoJ). - Persist the execution ledger and raw trial traces to durable S3 storage and upload a raw-traces artifact so evidence survives outside the ephemeral runner (Vu6-M). - Restore prior ledger and result directories before allowing workflow resumption so a rerun continues from the last complete pair boundary (Vu6-N). Each fix is covered by focused regression tests in the benchmark validation suite; all 492 benchmark validation tests pass. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
morluto
added a commit
that referenced
this pull request
Aug 2, 2026
Close the remaining gaps in the #328 review thread fixes: - Preserve PR changed paths and the base SHA through the forced-full (ci:benchmark-full) workflow branch so dataset version-bump invariants cannot be bypassed by the label; add planner and workflow regression tests for the force-full-with-base path (VuxoK). - Replace the vacuous compose-signature test with one that exercises _comparison_job directly: stripping only the c2 treatment overlay makes control/treatment jobs compare equal, while an unrelated sidecar survives normalization and is rejected (Vu6-O). - Add a build_observation_evidence test that records agent name, agent version, and model provider identities from trial data and fails closed when the observed agent version drifts from the frozen runtime snapshot (Vu6-S). All 510 benchmark validation and CI execution-policy tests pass. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ac8d400a94
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
morluto
added a commit
that referenced
this pull request
Aug 2, 2026
* benchmarks: add subset incidence determinant * benchmarks: bump agent-workflow-v1 suite version for planner gate Bump the agent-workflow-v1 dataset version from 1.1.0 to 1.2.0 to satisfy the suite version-bump planner gate introduced by PR #328. The dataset manifest and task digests are unchanged because the version field lives in suite.toml, not in the generated dataset.toml. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * benchmarks: sync agent-workflow-v1 README version to 1.2.0 Update the README version sentence from 1.1.0 to 1.2.0 to match the suite.toml version bump. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(benchmarks): reject boolean JSON numbers and enforce single-evidence limit The subset-incidence-determinant verifier treated JSON booleans as integers (True == 1, False == 0), so submissions replacing numeric fields with booleans passed the agent-visible enum/numeric contracts. Add an _is_int helper that rejects booleans and apply it to sample_n, mask_order, diagonal_weights, and every trace numeric field. Also enforce the maxItems: 1 evidence contract so a repeated descriptor cannot earn evidence_validity. Regenerate the task digest and Dockerfile checksum, and sync the gap-ledger/comparison-plan manifest digest. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * relax subset determinant prose contract * sync subset benchmark digests * Accept subset determinant sample variants --------- Co-authored-by: Grace Lee Rui Yue <graceleeruiyue@graces-air-2.lan> Co-authored-by: morluto <76467478+morluto@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why
The previous benchmark layer could render and run suites, but the held-out workflow still started Jacobian for the control, batched repetitions into condition-wide jobs, accepted caller-selected evidence classes, and recorded budgets without enforcing a comparison boundary. Verifier and Oracle entry files were bound individually, but their complete trees, the actual prompt, agent version, and runtime identity were not all frozen together.
Impact
Held-out experiments now fail closed unless every task/repetition has exactly one C1/C2 pair, every result is bound to the same frozen plan and complete ledger, core metrics and accounting are present, and all comparison invariants match. C1 cannot declare a Jacobian image, sidecar, or MCP server. C2 must match the digest-pinned image plus advertised server, catalog, and policy identities before any paid run starts.
Harbor 0.20 cannot hard-stop Codex at token or cost thresholds. The runner therefore checks after each complete pair; missing accounting or an overage marks the run INCOMPLETE and prevents a valid comparison.
Validation
make harbor-check— 477 passedmake lint— Ruff, formatting, and complexity passedmake check-staticreaches only the pre-existing macOSresource.prlimitmypy error insrc/jacobian/bounded_process.py:94The Docker-backed
make heldout-smokenop/Oracle phase was not runnable on this host because Docker is not installed; the same fixture's manifest, complete tree digests, renderer, and pair expansion passed without Docker.