TRT-2870: A/B test A (control): adversarial payload false-revert evals - #660
Closed
stbenjam wants to merge 11 commits into
Closed
TRT-2870: A/B test A (control): adversarial payload false-revert evals#660stbenjam wants to merge 11 commits into
stbenjam wants to merge 11 commits into
Conversation
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
|
Skipping CI for Draft Pull Request. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: stbenjam The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds point-in-time controls to the payload-analysis evaluation, timestamps cases 001–014 and 018–020, expands causal constraints, documents the case index, and synchronizes the CI plugin version to 0.0.78. ChangesPayload analysis evaluation
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 10✅ Passed checks (10 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
This was referenced Jul 30, 2026
stbenjam
changed the title
stbenjam
marked this pull request as ready for review
July 30, 2026 18:42
openshift-ci
Bot
removed
the
do-not-merge/work-in-progress
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (1)
plugins/ci/evals/cases/payload-analysis/case-010/annotations.yaml (1)
5-9: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winName the fourth failed job explicitly.
The notes claim four failed jobs but enumerate only three and replace the fourth with “one more.” Record the exact job identifier so the fixture remains auditable.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/ci/evals/cases/payload-analysis/case-010/annotations.yaml` around lines 5 - 9, Update the notes in the payload-analysis fixture to replace “one more” with the exact identifier of the fourth failed job, preserving the existing list of three jobs and the surrounding scenario description.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/ci/evals/cases/payload-analysis/case-001/annotations.yaml`:
- Around line 16-21: Extend the payload-analysis contract and its scoring logic
to recognize expected_ci_config_issues alongside expected_candidates. Update the
schema for eval-payload-analysis.yaml and the judge/scoring path so this field
is validated and evaluated, ensuring the annotated PR and expected failing job
are not ignored.
In `@plugins/ci/evals/cases/payload-analysis/case-002/annotations.yaml`:
- Around line 12-20: The expected_failed_job_count and expected_failing_jobs
entries in the case-002 annotations must describe the same failure set. Update
the relevant annotation values by either adding the five missing job IDs or
changing the count and associated notes to match the eight listed jobs,
preserving the intended evaluation behavior.
In `@plugins/ci/evals/cases/payload-analysis/case-003/annotations.yaml`:
- Around line 2-15: Update expected_failed_job_count in the payload-analysis
annotation to 3 so it matches the three entries listed under
expected_candidates[0].expected_failing_jobs: e2e-aws-ovn, e2e-azure-ovn, and
e2e-gcp-ovn.
In `@plugins/ci/evals/cases/payload-analysis/case-008/annotations.yaml`:
- Line 1: Update the expected_phase fixture value to one of the schema-supported
phases: Rejected, Ready, or Accepted, using the phase this case is intended to
assert. Only change the shared schema and scorer if this fixture intentionally
represents an unasserted phase, and then add explicit support for that behavior.
In `@plugins/ci/evals/cases/payload-analysis/case-018/annotations.yaml`:
- Around line 16-56: Update the payload-analysis evaluator to machine-check both
prohibited candidates and required causal evidence: for
plugins/ci/evals/cases/payload-analysis/case-018/annotations.yaml lines 16-56,
reject hypershift#8871 even when oc#2279 is also recommended; for case-019
annotations.yaml lines 8-50, require the interval-union explanation; and for
case-020 annotations.yaml lines 8-48, require the ordered infrastructure causal
chain. Ensure scoring fails when any forbidden candidate is present or a
required explanation is absent, including when expected_candidates is empty.
---
Nitpick comments:
In `@plugins/ci/evals/cases/payload-analysis/case-010/annotations.yaml`:
- Around line 5-9: Update the notes in the payload-analysis fixture to replace
“one more” with the exact identifier of the fourth failed job, preserving the
existing list of three jobs and the surrounding scenario description.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 0d62a194-3ccb-4824-9f46-ae63969ad474
📒 Files selected for processing (35)
plugins/ci/evals/cases/payload-analysis/README.mdplugins/ci/evals/cases/payload-analysis/case-001/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-001/input.yamlplugins/ci/evals/cases/payload-analysis/case-002/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-002/input.yamlplugins/ci/evals/cases/payload-analysis/case-003/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-003/input.yamlplugins/ci/evals/cases/payload-analysis/case-004/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-004/input.yamlplugins/ci/evals/cases/payload-analysis/case-005/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-005/input.yamlplugins/ci/evals/cases/payload-analysis/case-006/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-006/input.yamlplugins/ci/evals/cases/payload-analysis/case-007/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-007/input.yamlplugins/ci/evals/cases/payload-analysis/case-008/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-008/input.yamlplugins/ci/evals/cases/payload-analysis/case-009/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-009/input.yamlplugins/ci/evals/cases/payload-analysis/case-010/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-010/input.yamlplugins/ci/evals/cases/payload-analysis/case-011/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-011/input.yamlplugins/ci/evals/cases/payload-analysis/case-012/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-012/input.yamlplugins/ci/evals/cases/payload-analysis/case-013/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-013/input.yamlplugins/ci/evals/cases/payload-analysis/case-014/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-014/input.yamlplugins/ci/evals/cases/payload-analysis/case-018/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-018/input.yamlplugins/ci/evals/cases/payload-analysis/case-019/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-019/input.yamlplugins/ci/evals/cases/payload-analysis/case-020/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-020/input.yaml
Contributor
There was a problem hiding this comment.
Caution
Inline review comments failed to post. This is likely due to GitHub's internal server error or limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.
Actionable comments posted: 5
🧹 Nitpick comments (1)
plugins/ci/evals/cases/payload-analysis/case-010/annotations.yaml (1)
5-9: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winName the fourth failed job explicitly.
The notes claim four failed jobs but enumerate only three and replace the fourth with “one more.” Record the exact job identifier so the fixture remains auditable.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/ci/evals/cases/payload-analysis/case-010/annotations.yaml` around lines 5 - 9, Update the notes in the payload-analysis fixture to replace “one more” with the exact identifier of the fourth failed job, preserving the existing list of three jobs and the surrounding scenario description.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/ci/evals/cases/payload-analysis/case-001/annotations.yaml`:
- Around line 16-21: Extend the payload-analysis contract and its scoring logic
to recognize expected_ci_config_issues alongside expected_candidates. Update the
schema for eval-payload-analysis.yaml and the judge/scoring path so this field
is validated and evaluated, ensuring the annotated PR and expected failing job
are not ignored.
In `@plugins/ci/evals/cases/payload-analysis/case-002/annotations.yaml`:
- Around line 12-20: The expected_failed_job_count and expected_failing_jobs
entries in the case-002 annotations must describe the same failure set. Update
the relevant annotation values by either adding the five missing job IDs or
changing the count and associated notes to match the eight listed jobs,
preserving the intended evaluation behavior.
In `@plugins/ci/evals/cases/payload-analysis/case-003/annotations.yaml`:
- Around line 2-15: Update expected_failed_job_count in the payload-analysis
annotation to 3 so it matches the three entries listed under
expected_candidates[0].expected_failing_jobs: e2e-aws-ovn, e2e-azure-ovn, and
e2e-gcp-ovn.
In `@plugins/ci/evals/cases/payload-analysis/case-008/annotations.yaml`:
- Line 1: Update the expected_phase fixture value to one of the schema-supported
phases: Rejected, Ready, or Accepted, using the phase this case is intended to
assert. Only change the shared schema and scorer if this fixture intentionally
represents an unasserted phase, and then add explicit support for that behavior.
In `@plugins/ci/evals/cases/payload-analysis/case-018/annotations.yaml`:
- Around line 16-56: Update the payload-analysis evaluator to machine-check both
prohibited candidates and required causal evidence: for
plugins/ci/evals/cases/payload-analysis/case-018/annotations.yaml lines 16-56,
reject hypershift#8871 even when oc#2279 is also recommended; for case-019
annotations.yaml lines 8-50, require the interval-union explanation; and for
case-020 annotations.yaml lines 8-48, require the ordered infrastructure causal
chain. Ensure scoring fails when any forbidden candidate is present or a
required explanation is absent, including when expected_candidates is empty.
---
Nitpick comments:
In `@plugins/ci/evals/cases/payload-analysis/case-010/annotations.yaml`:
- Around line 5-9: Update the notes in the payload-analysis fixture to replace
“one more” with the exact identifier of the fourth failed job, preserving the
existing list of three jobs and the surrounding scenario description.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 0d62a194-3ccb-4824-9f46-ae63969ad474
📒 Files selected for processing (35)
plugins/ci/evals/cases/payload-analysis/README.mdplugins/ci/evals/cases/payload-analysis/case-001/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-001/input.yamlplugins/ci/evals/cases/payload-analysis/case-002/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-002/input.yamlplugins/ci/evals/cases/payload-analysis/case-003/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-003/input.yamlplugins/ci/evals/cases/payload-analysis/case-004/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-004/input.yamlplugins/ci/evals/cases/payload-analysis/case-005/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-005/input.yamlplugins/ci/evals/cases/payload-analysis/case-006/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-006/input.yamlplugins/ci/evals/cases/payload-analysis/case-007/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-007/input.yamlplugins/ci/evals/cases/payload-analysis/case-008/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-008/input.yamlplugins/ci/evals/cases/payload-analysis/case-009/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-009/input.yamlplugins/ci/evals/cases/payload-analysis/case-010/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-010/input.yamlplugins/ci/evals/cases/payload-analysis/case-011/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-011/input.yamlplugins/ci/evals/cases/payload-analysis/case-012/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-012/input.yamlplugins/ci/evals/cases/payload-analysis/case-013/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-013/input.yamlplugins/ci/evals/cases/payload-analysis/case-014/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-014/input.yamlplugins/ci/evals/cases/payload-analysis/case-018/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-018/input.yamlplugins/ci/evals/cases/payload-analysis/case-019/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-019/input.yamlplugins/ci/evals/cases/payload-analysis/case-020/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-020/input.yaml
🛑 Comments failed to post (4)
plugins/ci/evals/cases/payload-analysis/case-001/annotations.yaml (1)
16-21: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash set -euo pipefail rg -n 'expected_ci_config_issues' plugins/ci/evalsRepository: openshift-eng/ai-helpers
Length of output: 258
🏁 Script executed:
#!/usr/bin/env bash set -euo pipefail printf '\n== files ==\n' git ls-files 'plugins/ci/evals/**' | sed -n '1,200p' printf '\n== search for schema/field usage ==\n' rg -n 'expected_ci_config_issues|expected_candidates|notes|ci_config_issues' plugins/ci/evals -S printf '\n== likely contract file preview ==\n' if [ -f plugins/ci/evals/eval-payload-analysis.yaml ]; then cat -n plugins/ci/evals/eval-payload-analysis.yaml | sed -n '1,220p' fi printf '\n== annotations case preview ==\n' cat -n plugins/ci/evals/cases/payload-analysis/case-001/annotations.yaml | sed -n '1,220p'Repository: openshift-eng/ai-helpers
Length of output: 24770
🏁 Script executed:
#!/usr/bin/env bash set -euo pipefail printf '\n== all references ==\n' rg -n 'expected_ci_config_issues|expected_candidates|has_revert_candidates|force_accept_expected|annotations\.yaml|expected_failing_jobs' plugins/ci -S printf '\n== payload eval judge logic around candidate handling ==\n' cat -n plugins/ci/evals/eval-payload-analysis.yaml | sed -n '300,420p' printf '\n== payload case files that use only supported fields ==\n' for f in plugins/ci/evals/cases/payload-analysis/case-*/annotations.yaml; do echo "--- $f" sed -n '1,40p' "$f" done | sed -n '1,220p'Repository: openshift-eng/ai-helpers
Length of output: 23613
Add
expected_ci_config_issuesto the payload-analysis contract.
plugins/ci/evals/eval-payload-analysis.yamlonly definesexpected_candidatesandnotes, and the scoring logic only looks atexpected_candidates, so this CI-regression expectation will be ignored unless the schema/judge are updated.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/ci/evals/cases/payload-analysis/case-001/annotations.yaml` around lines 16 - 21, Extend the payload-analysis contract and its scoring logic to recognize expected_ci_config_issues alongside expected_candidates. Update the schema for eval-payload-analysis.yaml and the judge/scoring path so this field is validated and evaluated, ensuring the annotated PR and expected failing job are not ignored.plugins/ci/evals/cases/payload-analysis/case-002/annotations.yaml (1)
12-20: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash set -euo pipefail # Inspect the target fixture and nearby context FILE="plugins/ci/evals/cases/payload-analysis/case-002/annotations.yaml" echo "=== $FILE (numbered) ===" cat -n "$FILE" echo echo "=== Repository matches for the case and related fields ===" rg -n --hidden --glob '!**/.git/**' \ -e 'expected_failed_job_count|expected_failing_jobs|case-002|blocking jobs|same regression|payload-analysis' \ plugins/ci/evals/cases/payload-analysis -S echo echo "=== File list for the case directory ===" fd -a . plugins/ci/evals/cases/payload-analysis/case-002Repository: openshift-eng/ai-helpers
Length of output: 7148
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "=== case-002 input.yaml ===" cat -n plugins/ci/evals/cases/payload-analysis/case-002/input.yaml echo echo "=== payload-analysis README excerpt ===" cat -n plugins/ci/evals/cases/payload-analysis/README.md echo echo "=== Search for the annotation schema and any checks on expected_failing_jobs ===" rg -n --hidden --glob '!**/.git/**' \ -e 'expected_failing_jobs|expected_failed_job_count|force_accept_expected|expected_candidates|has_revert_candidates' \ plugins/ci/evals -S echo echo "=== Inspect similar cases with expected_failing_jobs lists ===" for f in \ plugins/ci/evals/cases/payload-analysis/case-001/annotations.yaml \ plugins/ci/evals/cases/payload-analysis/case-004/annotations.yaml \ plugins/ci/evals/cases/payload-analysis/case-006/annotations.yaml \ plugins/ci/evals/cases/payload-analysis/case-008/annotations.yaml \ plugins/ci/evals/cases/payload-analysis/case-014/annotations.yaml do echo "--- $f ---" cat -n "$f" echo doneRepository: openshift-eng/ai-helpers
Length of output: 19181
🏁 Script executed:
#!/bin/bash set -euo pipefail FILE="plugins/ci/evals/eval-payload-analysis.yaml" echo "=== Relevant schema and rubric sections from $FILE ===" sed -n '55,90p;300,375p' "$FILE"Repository: openshift-eng/ai-helpers
Length of output: 5926
Align the failure count with the listed jobs
plugins/ci/evals/cases/payload-analysis/case-002/annotations.yaml:12-20—expected_failed_job_countsays 13, but this candidate lists only 8 jobs. Add the missing 5 job IDs or update the count/notes to match the intended failure set.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/ci/evals/cases/payload-analysis/case-002/annotations.yaml` around lines 12 - 20, The expected_failed_job_count and expected_failing_jobs entries in the case-002 annotations must describe the same failure set. Update the relevant annotation values by either adding the five missing job IDs or changing the count and associated notes to match the eight listed jobs, preserving the intended evaluation behavior.plugins/ci/evals/cases/payload-analysis/case-003/annotations.yaml (1)
2-15: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Align
expected_failed_job_countwith the three expected failing jobs.The count is
2, butexpected_failing_jobslists three jobs and the notes describe AWS, Azure, and GCP failures. This can cause the evaluator to reject correct output.Suggested fix
-expected_failed_job_count: 2 +expected_failed_job_count: 3Also applies to: 18-20
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/ci/evals/cases/payload-analysis/case-003/annotations.yaml` around lines 2 - 15, Update expected_failed_job_count in the payload-analysis annotation to 3 so it matches the three entries listed under expected_candidates[0].expected_failing_jobs: e2e-aws-ovn, e2e-azure-ovn, and e2e-gcp-ovn.plugins/ci/evals/cases/payload-analysis/case-008/annotations.yaml (1)
1-1: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Use a schema-valid expected phase.
The evaluator contract defines
expected_phaseasRejected|Ready|Accepted, but this fixture sets it to"". Replace it with the actual valid phase, or update the shared schema and scorer to explicitly support an unasserted phase.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/ci/evals/cases/payload-analysis/case-008/annotations.yaml` at line 1, Update the expected_phase fixture value to one of the schema-supported phases: Rejected, Ready, or Accepted, using the phase this case is intended to assert. Only change the shared schema and scorer if this fixture intentionally represents an unasserted phase, and then add explicit support for that behavior.
Member
Author
|
Point-in-time eval integrity update:
|
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/ci/evals/eval-payload-analysis.yaml`:
- Around line 297-330: Update the URL and shell-command checks in the
payload-analysis evaluator so SHA-pinned immutable GitHub commit endpoints,
including pinned raw-content fetches, remain allowed regardless of whether they
use WebFetch, curl, wget, or gh api. Refine github_url() and the mutable_gh
patterns to flag only mutable or post-hoc GitHub lookups, while preserving
violations for unpinned pull requests, issues, comments, reviews, and other
mutable endpoints.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 67d1a101-a19f-41e6-be2a-1ed3ee7a7cd5
📒 Files selected for processing (38)
.claude-plugin/marketplace.jsondocs/index.htmlplugins/ci/.claude-plugin/plugin.jsonplugins/ci/evals/cases/payload-analysis/case-001/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-001/input.yamlplugins/ci/evals/cases/payload-analysis/case-002/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-002/input.yamlplugins/ci/evals/cases/payload-analysis/case-003/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-003/input.yamlplugins/ci/evals/cases/payload-analysis/case-004/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-004/input.yamlplugins/ci/evals/cases/payload-analysis/case-005/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-005/input.yamlplugins/ci/evals/cases/payload-analysis/case-006/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-006/input.yamlplugins/ci/evals/cases/payload-analysis/case-007/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-007/input.yamlplugins/ci/evals/cases/payload-analysis/case-008/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-008/input.yamlplugins/ci/evals/cases/payload-analysis/case-009/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-009/input.yamlplugins/ci/evals/cases/payload-analysis/case-010/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-010/input.yamlplugins/ci/evals/cases/payload-analysis/case-011/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-011/input.yamlplugins/ci/evals/cases/payload-analysis/case-012/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-012/input.yamlplugins/ci/evals/cases/payload-analysis/case-013/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-013/input.yamlplugins/ci/evals/cases/payload-analysis/case-014/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-014/input.yamlplugins/ci/evals/cases/payload-analysis/case-018/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-018/input.yamlplugins/ci/evals/cases/payload-analysis/case-019/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-019/input.yamlplugins/ci/evals/cases/payload-analysis/case-020/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-020/input.yamlplugins/ci/evals/eval-payload-analysis.yaml
🚧 Files skipped from review as they are similar to previous changes (6)
- plugins/ci/evals/cases/payload-analysis/case-009/input.yaml
- plugins/ci/evals/cases/payload-analysis/case-020/input.yaml
- plugins/ci/evals/cases/payload-analysis/case-011/input.yaml
- plugins/ci/evals/cases/payload-analysis/case-003/annotations.yaml
- plugins/ci/evals/cases/payload-analysis/case-019/annotations.yaml
- plugins/ci/evals/cases/payload-analysis/case-013/input.yaml
Contributor
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
plugins/ci/evals/eval-payload-analysis.yaml (1)
334-349: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winBind evidence checks to the exact case cutoff.
A command such as
git log --before=noworgh api .../commits?until=2099-01-01passes the trace gate, and the integrity judge treats any immutable code as admissible even if its commit was created after the cutoff.
plugins/ci/evals/eval-payload-analysis.yaml#L334-L349: parse absolute--before/--untilvalues and require them to be at or beforeanalysis_cutoff; reject relative dates.plugins/ci/evals/eval-payload-analysis.yaml#L507-L511: require immutable code evidence to establish that the referenced commit or artifact existed byanalysis_cutoff.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/ci/evals/eval-payload-analysis.yaml` around lines 334 - 349, Update the command checks around git_log_pattern and gh api commit searches to parse absolute --before/--until values, require them to be no later than analysis_cutoff, and reject relative dates such as “now”. In the immutable-code validation at plugins/ci/evals/eval-payload-analysis.yaml lines 507-511, require evidence that the referenced commit or artifact existed by analysis_cutoff before accepting it.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@plugins/ci/evals/eval-payload-analysis.yaml`:
- Around line 334-349: Update the command checks around git_log_pattern and gh
api commit searches to parse absolute --before/--until values, require them to
be no later than analysis_cutoff, and reject relative dates such as “now”. In
the immutable-code validation at plugins/ci/evals/eval-payload-analysis.yaml
lines 507-511, require evidence that the referenced commit or artifact existed
by analysis_cutoff before accepting it.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: d8165587-85f5-4ae2-8502-4144a031d856
📒 Files selected for processing (1)
plugins/ci/evals/eval-payload-analysis.yaml
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
plugins/ci/evals/eval-payload-analysis.yaml (1)
72-75: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDocument the
any_ofkey name.The schema text describes "a name and a list of acceptable evidence phrases" but the judge (Line 562) reads
claim["any_of"], and cases use that key. Naming it here keeps fixture authors from inventing a different key that silently disables the check.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/ci/evals/eval-payload-analysis.yaml` around lines 72 - 75, Update the `required_claims` schema description in the eval payload documentation to explicitly name the `any_of` key as the list of acceptable evidence phrases, matching the `claim["any_of"]` access used by the judge and existing cases.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/ci/evals/cases/payload-analysis/case-020/annotations.yaml`:
- Around line 19-22: Widen the “Azure DNS trigger” claim’s any_of phrase set in
the annotations configuration by adding tolerant variants covering “429
responses from Azure DNS” and throttled DNS queries with HTTP 429, while
preserving the existing phrases.
In `@plugins/ci/evals/eval-payload-analysis.yaml`:
- Around line 529-547: Replace the unguarded int() conversions for
min_confidence and confidence_score in the candidate evaluation and
forbidden_revert_candidates checks with defensive coercion that handles None,
numeric strings, and non-numeric values without raising. Ensure invalid values
produce a clean failure result rather than an exception, while preserving the
existing threshold and actionable-candidate checks.
- Around line 434-435: Update the Bash URL-scanning loop around
check_external_url to inspect only URLs used by fetch-like invocations, rather
than every URL present in command text. Exclude URLs in heredocs, echoed
content, grep operations, and other non-fetch contexts while preserving checks
for actual network fetchers such as curl or wget.
---
Nitpick comments:
In `@plugins/ci/evals/eval-payload-analysis.yaml`:
- Around line 72-75: Update the `required_claims` schema description in the eval
payload documentation to explicitly name the `any_of` key as the list of
acceptable evidence phrases, matching the `claim["any_of"]` access used by the
judge and existing cases.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 2f5cf463-8eff-4f1d-acdc-cafd37524215
📒 Files selected for processing (4)
plugins/ci/evals/cases/payload-analysis/case-018/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-019/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-020/annotations.yamlplugins/ci/evals/eval-payload-analysis.yaml
Comment on lines
+19
to
+22
| - name: "Azure DNS trigger" | ||
| any_of: | ||
| - "dns 429" | ||
| - "dns throttling" |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Widen this claim's phrase set.
case_constraints does plain lowercase substring matching, and this claim only accepts "dns 429" or "dns throttling". A correct report phrased as "429 responses from Azure DNS" or "throttled DNS queries (HTTP 429)" fails a min_pass_rate: 1.0 gate. The other claims here carry three alternatives each; add a couple more tolerant variants.
🛡️ Suggested additions
- name: "Azure DNS trigger"
any_of:
- "dns 429"
- "dns throttling"
+ - "throttled dns"
+ - "429"📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| - name: "Azure DNS trigger" | |
| any_of: | |
| - "dns 429" | |
| - "dns throttling" | |
| - name: "Azure DNS trigger" | |
| any_of: | |
| - "dns 429" | |
| - "dns throttling" | |
| - "throttled dns" | |
| - "429" |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/ci/evals/cases/payload-analysis/case-020/annotations.yaml` around
lines 19 - 22, Widen the “Azure DNS trigger” claim’s any_of phrase set in the
annotations configuration by adding tolerant variants covering “429 responses
from Azure DNS” and throttled DNS queries with HTTP 429, while preserving the
existing phrases.
Comment on lines
+434
to
+435
| for url in re.findall(r"https?://[^\s'\"<>]+", command, re.I): | ||
| check_external_url(url, "Bash") |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Scanning every URL in every Bash command will produce false-positive hard failures.
re.findall over the whole command matches URLs that are never fetched — e.g. a heredoc writing the HTML/YAML report (which legitimately contains https://github.com/openshift/api/pull/2920), a grep of snapshot artifacts, or an echo. Since point_in_time_trace_hygiene has min_pass_rate: 1.0 (Line 781), one such command fails the whole run even though no mutable lookup occurred. Restrict the scan to fetch-like invocations.
🐛 Proposed fix: only inspect URLs passed to fetchers
- for url in re.findall(r"https?://[^\s'\"<>]+", command, re.I):
- check_external_url(url, "Bash")
+ for fetch in re.finditer(
+ r"\b(?:curl|wget|http|https)\b[^\n;&|]*", command, re.I
+ ):
+ for url in re.findall(
+ r"https?://[^\s'\"<>]+", fetch.group(0), re.I
+ ):
+ check_external_url(url, "Bash")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/ci/evals/eval-payload-analysis.yaml` around lines 434 - 435, Update
the Bash URL-scanning loop around check_external_url to inspect only URLs used
by fetch-like invocations, rather than every URL present in command text.
Exclude URLs in heredocs, echoed content, grep operations, and other non-fetch
contexts while preserving checks for actual network fetchers such as curl or
wget.
Comment on lines
+529
to
+547
| minimum = int(expected.get("min_confidence", 0)) | ||
| actual = int(candidate.get("confidence_score", 0)) | ||
| if actual < minimum: | ||
| failures.append( | ||
| f"Expected candidate {url} scored {actual}, below {minimum}" | ||
| ) | ||
|
|
||
| for url in case_data.get("forbidden_revert_candidates", []) or []: | ||
| normalized = str(url).rstrip("/").lower() | ||
| candidate = by_url.get(normalized) | ||
| if candidate is None: | ||
| continue | ||
| confidence = int(candidate.get("confidence_score", 0)) | ||
| eligible = candidate.get("revert_eligible") is True | ||
| if confidence >= 85 or eligible: | ||
| failures.append( | ||
| f"Forbidden revert candidate {normalized} remained actionable " | ||
| f"(confidence={confidence}, revert_eligible={eligible})" | ||
| ) |
Contributor
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Unguarded int() casts can raise inside the judge.
confidence_score / min_confidence may be None, "92", or a non-numeric string; int(None) and int("high") raise instead of returning a clean (False, ...) result. Coerce defensively so the gate reports a failure rather than erroring out.
🛡️ Proposed fix
+ def as_score(value, default=0):
+ try:
+ return int(float(value))
+ except (TypeError, ValueError):
+ return default
+
for expected in case_data.get("expected_candidates", []) or []:
@@
- minimum = int(expected.get("min_confidence", 0))
- actual = int(candidate.get("confidence_score", 0))
+ minimum = as_score(expected.get("min_confidence"))
+ actual = as_score(candidate.get("confidence_score"))
@@
- confidence = int(candidate.get("confidence_score", 0))
+ confidence = as_score(candidate.get("confidence_score"))📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| minimum = int(expected.get("min_confidence", 0)) | |
| actual = int(candidate.get("confidence_score", 0)) | |
| if actual < minimum: | |
| failures.append( | |
| f"Expected candidate {url} scored {actual}, below {minimum}" | |
| ) | |
| for url in case_data.get("forbidden_revert_candidates", []) or []: | |
| normalized = str(url).rstrip("/").lower() | |
| candidate = by_url.get(normalized) | |
| if candidate is None: | |
| continue | |
| confidence = int(candidate.get("confidence_score", 0)) | |
| eligible = candidate.get("revert_eligible") is True | |
| if confidence >= 85 or eligible: | |
| failures.append( | |
| f"Forbidden revert candidate {normalized} remained actionable " | |
| f"(confidence={confidence}, revert_eligible={eligible})" | |
| ) | |
| def as_score(value, default=0): | |
| try: | |
| return int(float(value)) | |
| except (TypeError, ValueError): | |
| return default | |
| for expected in case_data.get("expected_candidates", []) or []: | |
| minimum = as_score(expected.get("min_confidence")) | |
| actual = as_score(candidate.get("confidence_score")) | |
| if actual < minimum: | |
| failures.append( | |
| f"Expected candidate {url} scored {actual}, below {minimum}" | |
| ) | |
| for url in case_data.get("forbidden_revert_candidates", []) or []: | |
| normalized = str(url).rstrip("/").lower() | |
| candidate = by_url.get(normalized) | |
| if candidate is None: | |
| continue | |
| confidence = as_score(candidate.get("confidence_score")) | |
| eligible = candidate.get("revert_eligible") is True | |
| if confidence >= 85 or eligible: | |
| failures.append( | |
| f"Forbidden revert candidate {normalized} remained actionable " | |
| f"(confidence={confidence}, revert_eligible={eligible})" | |
| ) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/ci/evals/eval-payload-analysis.yaml` around lines 529 - 547, Replace
the unguarded int() conversions for min_confidence and confidence_score in the
candidate evaluation and forbidden_revert_candidates checks with defensive
coercion that handles None, numeric strings, and non-numeric values without
raising. Ensure invalid values produce a clean failure result rather than an
exception, while preserving the existing threshold and actionable-candidate
checks.
Contributor
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
plugins/ci/evals/eval-payload-analysis.yaml (3)
527-555: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winEnforce expected failing-job linkage in the deterministic judge.
The schema and LLM prompt require each expected candidate to explain its
expected_failing_jobs, but this loop only checks URL presence and confidence. A candidate with the correct PR and score but an empty or unrelatedfailing_jobslist therefore passes the hard gate. Compare the normalized expected and actual job sets before accepting the candidate.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/ci/evals/eval-payload-analysis.yaml` around lines 527 - 555, Update the expected_causes validation loop to also enforce expected_failing_jobs linkage. Normalize the expected job names and the candidate’s failing_jobs values into comparable sets, and append a failure when they differ before accepting the candidate; retain the existing URL and minimum-confidence checks.
374-388: 🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy liftDo not whitelist payload-agent reports by filename alone.
is_completed_agent_report()accepts any matching artifact URL without checking the payload identity or whether the report was generated byanalysis_cutoff. A later agent report can contain exactly the post-cutoff reverts or outcomes thatpoint_in_time_integrityis intended to reject. Reject these URLs by default, or require verifiable artifact metadata proving temporal and payload scope.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/ci/evals/eval-payload-analysis.yaml` around lines 374 - 388, Update is_completed_agent_report to stop accepting completed report URLs based only on their filename and path; reject them by default unless verifiable artifact metadata confirms the expected payload identity and analysis_cutoff scope. Ensure check_external_url continues recording such URLs as violations when that metadata cannot be established.
724-738: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winRender
outputs.modified_fileshereThis prompt only renders
outputs.files, but the eval already treatsoutputs["files"]andoutputs["modified_files"]as the source of truth. If the YAML or HTML lands inmodified_files, the judge misses the report entirely. Render both mappings here.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/ci/evals/eval-payload-analysis.yaml` around lines 724 - 738, Update the report-rendering sections in the eval prompt to iterate over both outputs.files and outputs.modified_files when selecting YAML payload results and HTML summary reports. Preserve the existing path filters and display format, while ensuring files present only in modified_files are included.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@plugins/ci/evals/eval-payload-analysis.yaml`:
- Around line 527-555: Update the expected_causes validation loop to also
enforce expected_failing_jobs linkage. Normalize the expected job names and the
candidate’s failing_jobs values into comparable sets, and append a failure when
they differ before accepting the candidate; retain the existing URL and
minimum-confidence checks.
- Around line 374-388: Update is_completed_agent_report to stop accepting
completed report URLs based only on their filename and path; reject them by
default unless verifiable artifact metadata confirms the expected payload
identity and analysis_cutoff scope. Ensure check_external_url continues
recording such URLs as violations when that metadata cannot be established.
- Around line 724-738: Update the report-rendering sections in the eval prompt
to iterate over both outputs.files and outputs.modified_files when selecting
YAML payload results and HTML summary reports. Preserve the existing path
filters and display format, while ensuring files present only in modified_files
are included.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 570fe4e7-0aa7-4c31-9c0f-663ca33acab2
📒 Files selected for processing (8)
plugins/ci/evals/cases/payload-analysis/case-001/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-002/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-003/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-008/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-010/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-012/annotations.yamlplugins/ci/evals/cases/payload-analysis/case-013/annotations.yamlplugins/ci/evals/eval-payload-analysis.yaml
…evert-evals # Conflicts: # .claude-plugin/marketplace.json # docs/index.html # plugins/ci/.claude-plugin/plugin.json
Member
Author
|
/test eval-payload-analysis |
2 tasks
stbenjam
changed the title
openshift-ci-robot
added
the
jira/valid-reference
|
@stbenjam: This pull request references TRT-2870 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci
Bot
added
the
needs-rebase
|
PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Member
Author
|
Closing this experiment PR. The hard-case A/B/C evals are very expensive to run (~50% increase in payload agent eval cost) and Opus doesn't succeed on them regardless, so the signal-to-cost ratio isn't there right now. Will revisit when better models are available that can handle these cases. |
This was referenced Aug 3, 2026
This was referenced Aug 10, 2026
stbenjam
added a commit
to stbenjam/ai-helpers
that referenced
this pull request
Aug 17, 2026
Address review feedback on the revived payload false-revert eval cases:
- Drop the excessive adversarial annotation fields (required_claims,
must_not_conclude, forbidden_revert_candidates, distractors,
key_evidence, discriminating_signal) from cases 018/019/020. Nothing
in the payload-analysis eval consumed them; the core false-positive
check is already covered by expected_candidates: [].
- Keep the point-in-time metadata (payload_completed_at in input,
analysis_cutoff in annotations) and extend it to the existing cases
001-014, since this is the highest-value part — it lets the eval
detect hindsight/cheating.
- Pass the cutoff to the skill as `--as-of {payload_completed_at}` and
instruct the agent to perform a strict point-in-time analysis.
- Add the point_in_time_integrity LLM judge (min_mean 4.0) that scores
evidence provenance against analysis_cutoff and flags post-hoc
leakage (later reverts, subsequent payload outcomes, present-day PR
state). Cases without a cutoff score max, so it is a no-op for any
future non-point-in-time case.
Deliberately left behind openshift-eng#660's programmatic trace-hygiene judge (heavy
and brittle) and the case_constraints adversarial-field judge.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
stbenjam
added a commit
that referenced
this pull request
Aug 21, 2026
…udge (replaces #660) (#701) * test(ci): add adversarial payload false-revert eval cases Revives the three evidence-heavy payload-analysis eval cases from #660 on top of current main. These are the control-arm cases capturing payloads where the Payload Agent recommended incorrect reverts, and they exercise the skill's ability to distinguish well-supported revert candidates from false attributions. - case-018: mixed true/false attribution — keep openshift/oc#2279 while rejecting the cross-tenant etcd evidence attributed to openshift/hypershift#8871. - case-019: reject openshift/api#2920 and #2923; the apparent long operator waits come from disjoint interval arithmetic in the Origin monitor test. - case-020: reject openshift/ovn-kubernetes#3298 and openshift/cloud-provider-azure#164; reconstruct the ordered GCP/Azure infrastructure chains and distinguish triggers from amplifiers, detectors, and cleanup fallout. Cases are registered in the eval README case index. docs/index.html is regenerated via `make update` to clear pre-existing plugin-metadata drift so the strict plugin lint passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(ci): bump ci plugin to 0.0.86 for new eval cases The check-version-bump CI gate requires a version bump for any change under plugins/ci/. Bump the plugin version and re-sync marketplace.json and docs/index.html via `make update`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(ci): add point-in-time cutoff + integrity judge, trim case fields Address review feedback on the revived payload false-revert eval cases: - Drop the excessive adversarial annotation fields (required_claims, must_not_conclude, forbidden_revert_candidates, distractors, key_evidence, discriminating_signal) from cases 018/019/020. Nothing in the payload-analysis eval consumed them; the core false-positive check is already covered by expected_candidates: []. - Keep the point-in-time metadata (payload_completed_at in input, analysis_cutoff in annotations) and extend it to the existing cases 001-014, since this is the highest-value part — it lets the eval detect hindsight/cheating. - Pass the cutoff to the skill as `--as-of {payload_completed_at}` and instruct the agent to perform a strict point-in-time analysis. - Add the point_in_time_integrity LLM judge (min_mean 4.0) that scores evidence provenance against analysis_cutoff and flags post-hoc leakage (later reverts, subsequent payload outcomes, present-day PR state). Cases without a cutoff score max, so it is a no-op for any future non-point-in-time case. Deliberately left behind #660's programmatic trace-hygiene judge (heavy and brittle) and the case_constraints adversarial-field judge. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(ci): support --as-of point-in-time cutoff in payload-analysis skill Add the --as-of TIMESTAMP flag to the payload-analysis argument contract and bound every external lookup and subagent investigation to the cutoff. Under --as-of the skill reasons only from evidence that existed when the payload completed: it ignores post-cutoff reverts, comments, and payload outcomes, caps the step-registry commit window at the cutoff, and never treats a later revert (or its absence) as causal evidence. This keeps the point-in-time eval from leaking post-cutoff signal into recommendations. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(ci): unify cutoff field name, add deterministic outcome judges Unify the point-in-time cutoff under one name: annotations.yaml now carries payload_completed_at, identical to input.yaml, instead of a differently-named analysis_cutoff. The duplication itself stays because the harness only exposes annotations.yaml to judges; the dataset schema now documents that constraint. Add deterministic check judges for the outcomes that were previously only graded by LLM rubric: expected_candidates_found (every expected candidate at/above min_confidence), no_unexpected_reverts (a hard false-revert gate — no non-RPM candidate at/above the revert threshold outside the expected set), failed_job_count_matches (tolerance 1), force_accept_matches, and expected_phase_matches. All gate at min_pass_rate 1.0, and the conditional ones use the harness if: field so inapplicable cases are skipped rather than auto-scored. Judge fixes: revert_scoring_accuracy had drifted from the skill (it described a retired 130-point rubric with a "single candidate" signal); it now attaches SKILL.md as context and judges against the current rubric, with a two-sided scale where a false revert scores 1 instead of falling through anchors written for the true-positive case. point_in_time_integrity uses if: instead of instructing the model to return max score for non-cutoff cases. Case fixes: case-008 records its real phase (Rejected in every historical run) instead of ""; case-010 notes now state the verified Insights API Gateway HTTP 500 story with all four job names; cases 010 and 012 declare expected_candidates: [] explicitly. Also enable parallelism: 3 and tag MLflow runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ci): address CodeRabbit findings on --as-of and eval judges SKILL.md: Step 3.6 now defines a single until_timestamp (the earlier of the window end and the --as-of cutoff) passed whole to both step-registry queries, replacing the until=<until_date>T23:59:59Z templates that would mangle a mid-day cutoff or silently re-admit same-day post-cutoff commits. Step 6.3 requests createdAt and treats post-cutoff revert state as unavailable: a pre-cutoff revert that merged after the cutoff counts as still Open. Eval: expected_candidates_found now verifies expected_failing_jobs linkage (substring match, short names vs full periodic names); yaml_results_valid requires the canonical candidate type field and only demands pr_url for non-RPM candidates; the outputs schema documents type-specific candidate fields. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(ci): progressive disclosure + report template for payload-analysis Restructure the payload-analysis skill for progressive disclosure: - Drop the "Required Skills" preamble that force-loaded the payload-results-yaml and payload-autodl-json schema skills before any work began; each is now loaded via the Skill tool at its point of use (Steps 6.5 and 8), keeping early context lean. - Move the three run-once blobs out of SKILL.md into references/ loaded at the step that needs them: the Step 4 investigation-subagent prompt (investigation-subagent.md), the report content rules (report-guide.md), and the Step 9 completeness-review prompt (completeness-review.md). The scoring rubric stays in SKILL.md — it is per-run core and the eval's revert_scoring_accuracy judge attaches SKILL.md as rubric context. - Replace ~220 lines of inline HTML fragments and partial CSS with assets/report-template.html — a complete fill-in-the-blanks page (placeholders plus BEGIN/END conditional and repeatable blocks) that Step 7 copies and fills. The old prose said "follow the styling conventions of the existing report format", which every run reinterpreted; the template is now the single source of structure and styling, so reports come out consistent across runs. New design: phase hero, stat tiles, confidence meters with a Confidence column in the reverts table, status pills, and per-payload history cells (status always encoded as text or luminance alongside color). SKILL.md drops from 882 lines / 10.1k words to 581 lines / 7.6k words. Bump ci to 0.0.87; marketplace + docs synced via make update. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(ci): report template mirrors the production Payload Agent format Replace the experimental template design with the canonical production report format (structure and CSS taken verbatim from a real claude-payload-agent run), parameterized with placeholders and BEGIN/END conditional blocks: executive summary with per-job persistence, revert verdict with the Score column and itemized rationale, no-revert and force-accept variants, blocking-jobs summary with S/F history patterns, per-job collapsible details with candidates-table/candidates-none alternatives, RHCOS changes with RPM candidates and per-hop diffs, informing tests, and adversarial review. Production runs had already drifted between each other on CSS details; the template pins the format so every run renders identically. Sync references/report-guide.md and the Step 10 checklist to the same section list. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(ci): correct case-011 expected_failed_job_count to 2 The payload had two failed blocking jobs (aggregated-aws-ovn-upgrade-5.0-major and aggregated-gcp-ovn-upgrade-5.0-micro), both from the external Insights API Gateway HTTP 500 incident. Confirmed by two independent eval runs (Codex/Harbor and a local claude-opus-4-6 run); the old count of 1 was annotation error and cost both LLM judges points against a correct analysis. Notes now name the jobs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ci): verify revert-threshold evidence and add claim audit to review Three changes aimed at the recurring false-revert signature (four production incidents, all scoring 85-95 from correlation stacks that were never evidence-checked): - Step 6.1: the self-skepticism re-verification now runs for ANY score at or above the revert threshold (>= 85), not only on cap overflow. Every historical false revert scored 85-95 — below the old trigger. - Rubric: the error-message-match +20-30 tier now requires an observed artifact linking the failing operation to the modified code (stack frame, log line, event); shared subsystem vocabulary explicitly does not qualify and scores +10. - Step 9: the completeness reviewer gains a claim-audit pass for candidates >= 85. Scores change through exactly one mechanism: striking an itemized signal by citing the artifact evidence that contradicts it, then recomputing mechanically. Speculative objections remain inadmissible — preserving the guard against the earlier failure mode where valid reverts were downgraded on flimsy doubts. Eval config: declare score_range on the three numeric judges and move skill under execution per harness deprecation. Bump ci to 0.0.88. Gitignore /eval/ and /tmp/ local harness output. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(ci): add no_answer_access cheating detector to payload-analysis eval Deterministic judge that fails a case when the agent transcript touches annotations.yaml, traverses the eval dataset directory, or echoes annotation-only vocabulary (expected_candidates, force_accept_expected, ...). Verified retroactively clean against all 17 CI case transcripts from the PR head run and every local Luna/Opus run; the harness never stages annotations into the workspace, so any hit means the agent went looking. Scans the main transcript; separate subagent transcript files are out of reach. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(ci): narrow cheating detector to path patterns only Annotation field-name patterns could false-positive when the agent legitimately reasons about its output schema; the two path patterns (annotations.yaml, evals/cases/payload-analysis) are sufficient tells. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(ci): cheating detector scans merged events incl. subagent transcripts collect.py merges subagents/*.jsonl into per-case events.json, so scanning the serialized events covers subagent tool calls and text. Raw stdout remains the fallback when no events were captured. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * revert(ci): restore payload-analysis skill to pre-audit behavior Restores SKILL.md and completeness-review.md to the exact content the passing CI run (2089641369093017600, tested 572a6f3) executed. The unvalidated behavior changes from 8094b81 (>= 85 re-verification, error-match tier tightening, Step 9 claim audit) went out with no eval evidence, and their first measured run (2089697973024854016) regressed case-007: the claim audit stamped the historical wrong answer (cluster-version-operator#1309 @ 90) as verified while the true cause (operator-framework-olm#1256) went unscored. Measurement-side changes are kept: all judges including the no_answer_access cheating detector, annotation fixes, and .gitignore guards. Future skill behavior changes need control-vs-treatment eval evidence with repeats before merging. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ci): correct payload-analysis ground truth for case-019 and case-020 case-019 was mislabeled as a false-revert trap with no expected candidate. The aws/azure 4.22->5.0 major-upgrade jobs fail because CVO #1427 (OTA-1997) added a Deployment-manifest template field the 4.22 CVO cannot render, so it silently skips its own Deployment during upgrade. This was quick-reverted by #1431 (TRT-2842) and re-landed as #1433, confirming #1427 as the real cause. Set has_revert_candidates=true and add #1427 (@100, aws/azure-ovn-upgrade) as the expected candidate; the api#2920/#2923 attributions remain red herrings. case-020 met all three Step 6.4 force-accept criteria — both blocking failures are temporary infrastructure (Azure 429 throttling / OSProvisioningTimedOut, GCP transient VIP reachability loss), no more than 2 blocking jobs, and 48.4h since the last accepted payload (>= 18h). Set force_accept_expected=true. Also normalize expected_failing_jobs across cases to semantic-core tokens for bidirectional substring matching, make the expected_candidates_found matcher match when either the annotation token or the reported job name contains the other, and bump the judge model to claude-opus-4-8. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(ci): raise payload-analysis LLM thresholds to 4.5 Tighten the quality bars now that the corrected ground truth produces consistently high scores: analysis_quality min_mean 3.5 -> 4.5 and revert_scoring_accuracy min_mean 3.0 -> 4.5. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(ci): make required_skill_invocations judge harness-neutral Detect skill usage via either signal so the one payload-analysis eval config scores both Claude Code and Codex runs: - Signal A (Claude Code): a normalized `Skill` tool invocation, read from the merged event stream with a raw stdout JSONL fallback. - Signal B (Codex and any harness without a Skill tool): the skill's canonical SKILL.md H1 heading appears in the transcript, meaning the skill body was loaded and run. Codex has no `Skill` tool, so the prior stdout-JSONL parse always failed there. Runner selection stays CLI-overridable (--agent codex --effort xhigh --model ...) over the claude-code defaults; no second config. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(ci): add no_model_refusal deterministic judge to payload-analysis Detect API-level model refusals (the model's safeguards flagged the request) in the agent or any subagent, and fail the case explicitly so the root cause is visible instead of surfacing downstream as a confusing "missing output files" failure. This is an environment/model signal, not a skill-quality one. Detection is structural to avoid false positives: - system record whose subtype starts with model_refusal (survives in both raw stdout stream-json and the flat merged events schema); - a synthetic assistant turn (model "<synthetic>") paired with stop_reason "refusal" on the same record — which excludes the <synthetic> 429 rate-limit record (stop_reason "stop_sequence"); - "safeguards flagged" text as corroboration only, never a standalone trigger. Reuses the no_answer_access corpus idiom (json.dumps(events) with a stdout fallback) and reports which corpus was scanned. Claude Code only for now: when the capture is not a Claude Code run (no system/init signature) the judge abstains (no score) rather than emit a misleading pass — Codex refusal capture shape is a TODO. Gated at min_pass_rate 1.0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ci): resolve payload-analysis dataset.path from config dir The harness resolves dataset.path against the config file's directory (EvalConfig.resolve_path uses config_dir = plugins/ci/evals), so the repo-root-relative value doubled to plugins/ci/evals/plugins/ci/evals/cases/payload-analysis and did not exist — workspace setup would error and score.py would load zero annotations. Use the config-dir-relative form (cases/payload-analysis), matching the majority of eval configs in the repo. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: regenerate docs/index.html after upstream merge Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ci): pass expected_candidates_found on explicit no-revert cases The judge gated on `if: annotations.get('expected_candidates')`, but an explicit empty list — the documented way to declare "no revert expected" — is falsy in Python, so the judge silently skipped (n/a) instead of passing on all five no-revert cases (009, 010, 011, 012, 020). Its check body already returns a pass for an empty expectation, so gate on presence (`is not None`) like failed_job_count_matches does. The false-positive direction remains enforced by no_unexpected_reverts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(ci): allow Codex network egress in payload-analysis eval The default workspace-write sandbox blocks DNS/network entirely, starving the snapshot-backed analysis of the live GCS/Prow, GitHub, and Sippy lookups it still needs. Enable network_access on the write sandbox via runner.settings, which only the Codex runner reads — the claude-code runner ignores it, so its behavior is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(ci): keep codex scratch off /tmp; score implicit no-revert cases Two payload-analysis eval changes: - Add a system-prompt line telling the agent (and, via relay, its subagents) to use $TMPDIR for temporary files and downloaded artifacts instead of /tmp, so codex runs don't fill the RAM-backed tmpfs. The top-level agent honors this; subagent propagation is best-effort since codex child processes don't inherit runner.system_prompt. - Broaden expected_candidates_found so a case declaring no-revert implicitly (has_revert_candidates: false with expected_candidates omitted) is scored vacuously, matching the explicit [] form. Skipped only when a case defines neither expectation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Stephen Benjamin <stbenjam+ai@redhat.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR: Adds 3 new cases where the payload agent was wrong. A/B testing current skills vs changes to try to make Opus solve them.
Control arm for A/B testing
This PR adds three evidence-heavy payload-analysis eval cases without changing the payload-analysis or payload-snapshot skills. It is the control arm for comparison with #659, which contains the identical eval dataset plus causal-signature and revert-gate improvements.
The cases are intentionally named only
case-018,case-019, andcase-020because agent-eval-harness exposes directory names to the model. This PR also removes answer leakage from all existing payload-analysis case directory names and adds a maintainer-only case index.New cases
case-018: mixed true/false attribution. Keep the well-supportedopenshift/oc#2279candidate while rejecting the cross-tenant etcd evidence attribution toopenshift/hypershift#8871.case-019: rejectopenshift/api#2920and#2923; the apparent long operator waits come from disjoint interval arithmetic in the Origin monitor test.case-020: rejectopenshift/ovn-kubernetes#3298andopenshift/cloud-provider-azure#164; reconstruct the ordered GCP/Azure infrastructure chains and distinguish triggers from amplifiers, detectors, and cleanup fallout.Failed Payload Agent analyses
Validation
case-NNNidentifiers.The historical snapshot-data PR will be linked here once its large data push completes.
Summary by CodeRabbit