feat(benchmarks): audit research status evidence - #309
Conversation
1478376 to
3f9da9a
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3f9da9a925
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3f9da9a925
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…idence, scope, and assurance A mathematically correct submission with escaped evidence, wrong scope, or wrong assurance previously earned the 0.7 base partial reward. Gate the base on all three components so only the 0.1 per-component scores remain when any component fails. Add focused attack tests for wrong scope, escaped evidence, and wrong assurance that assert the base reward is withheld while the metric fields remain independently reported. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ed440193c5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
d8aa1f1 to
d0efc8c
Compare
…est for research-status-evidence-audit The verifier.py reward-gating change invalidated the Dockerfile checksum label and the dataset.toml task digest. Regenerate both via sync_harbor_verifier_support --write and check_harbor_dataset --write, and update the manifest digest in the comparison-plan and gap-ledger to match the regenerated dataset.toml. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
d0efc8c to
71072a5
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 71072a5028
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Address all unresolved review threads on PR #309: - Validate the immutable frozen input from /tests/input.json instead of the agent-writable /app/input.json copy; reject any byte mismatch so a tampered citation packet cannot score (PRRT_kwDOThEfjc6Vuw0q). - Bind the evidence explanation to the submitted result via a RESULT_JSON marker, preventing keyword-only evidence from scoring without supporting the classified cases (PRRT_kwDOThEfjc6Vuw0u). - Catch UnicodeError alongside OSError when reading evidence so invalid UTF-8 yields a zero-scored rejection instead of a crash (PRRT_kwDOThEfjc6Vuw0y). - Require the exact expected citation set per case instead of a subset, rejecting unrelated selected citations like SW1999 for the binomial case (PRRT_kwDOThEfjc6Vu4rI). - Enforce a 1 MiB evidence size bound before hashing and decoding so oversized malformed evidence cannot OOM or time out the verifier (PRRT_kwDOThEfjc6Vu4rK). - Reject undeclared result fields by requiring set(result) == {"cases"}, enforcing the schema's additionalProperties: false boundary (PRRT_kwDOThEfjc6Vu4rL). - Validate that case_id is a string before set membership, preventing TypeError on unhashable JSON values (PRRT_kwDOThEfjc6Vu8lt). - Publish the unsupported_inference vocabulary as an enum in the submission schema so agents can match the Oracle's serialization (PRRT_kwDOThEfjc6Vu8lu). - Record the unsupported exponent-range characterization inference for the Lebesgue-Nagell case, reflecting that the frozen citations do not substantiate the "small and sufficiently large" range description (PRRT_kwDOThEfjc6Vu8lv). - Refresh the verifier checksum and dataset manifest digest (PRRT_kwDOThEfjc6Vu8ls). - Make tests/input.json byte-identical to environment/input.json so the frozen-input comparison works in the test harness. Add regression tests for tampered input, extra citations, undeclared result fields, non-string case IDs, missing RESULT_JSON marker, oversized evidence, invalid UTF-8, and the required exponent-range inference. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b9937df148
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 203302ec6d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 79d0e9585a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 51280018c5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7841100fb1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The RESULT_JSON: marker carries the classifications, case IDs, and selected evidence IDs, which already encode the resolution, partial-progress, historical-openness, and problem-listing terms. Building the semantic text from the whole file let a submission whose prose supported none of the classifications (e.g. "This is not a problem statement.") pass every semantic_claims check and earn full evidence validity. Construct the lowered semantic text from prose only, so the result marker can no longer supply the explanation-bound terms. Add an attack test that asserts a canonical result marker with non-supporting prose is rejected for evidence validity while the metric fields remain independently reported. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 949bee59c4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 058ff8b9aa
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| semantic_claims = ( | ||
| ( | ||
| any(term in lowered for term in ("resolved", "solves", "all integer solutions")) | ||
| and any(term in lowered for term in ("binomial", "bmss2019")) | ||
| ), |
There was a problem hiding this comment.
Fresh evidence after the earlier explanation-binding comment: the new semantic_claims checks still test only for substring presence and ignore negation. A canonical result paired with prose such as “the BMSS2019 binomial is not resolved; KP2025's Lebesgue results are not partial; Z2008 is not a historical open problem about vanishing; W2007 is not a workshop problem listing” satisfies all four checks and receives full evidence validity and reward even though every classification is contradicted. Validate affirmative claims rather than keyword co-occurrence.
AGENTS.md reference: AGENTS.md:L18-L19
Useful? React with 👍 / 👎.
morluto
changed the title
2 participants
Summary
Adds one Hard, literature-grounded Harbor benchmark that audits whether frozen evidence actually supports research-status claims without asking the agent to solve an open problem.
Source and provenance
amphora/ResearchMath-14kf22d0f28b55e6e777acf82e722d97ae982dff02etestrows 0, 1, 2, and 56a4f30bc54142ae03756a412b1c54a2854e81e1d22f8311dc7aa0ec7796e26e0Selection and quality
Independent score: 87/100. The task adds citation-to-claim support auditing, chronology-sensitive status calibration, and explicit separation between historical evidence, partial progress, reported resolution, and unsupported present-day status. It was checked against merged tasks and active benchmark PRs at the workflow, verifier, failure-mode, and assurance-boundary levels.
Nearby candidates were rejected when their current status could not be independently adjudicated from frozen evidence or when they would duplicate the same evidence-calibration workflow.
Verifier design
The clean-room verifier checks:
COMPUTEDassurance boundaries.It does not assert the truth of an open conjecture or claim that frozen historical citations establish current status.
Validation
git diff --check: passedNative macOS mypy still encounters the repository's known
resource.prlimitplatform-stub issue; the Linux target used by CI passes. A Docker-backed Harbor run was unavailable on this host, so it is not claimed; Oracle and adversarial scoring are exercised through the repository test harness.Reusable-tool review
This implementation exposed a possible future citation-support/status-audit helper, but only one validated reuse episode exists. Per the tool-quality gate, no premature tool or skill PR is included; the gap remains deferred until repeated benchmark implementations establish a stable reusable contract.