feat(jira): add agent-eval-harness eval for jira:create skill - #615
Draft
cblecker wants to merge 1 commit into
Draft
feat(jira): add agent-eval-harness eval for jira:create skill#615cblecker wants to merge 1 commit into
cblecker wants to merge 1 commit into
Conversation
Add eval definition and 14 test cases covering all 6 issue types (story, bug, epic, feature, task, feature-request), all active convention files (CNTRLPLANE, OCPBUGS, GCP-HCP, HyperShift), and validation edge cases (bad summary anti-pattern, credential detection). Uses dry-run approach: system prompt instructs the skill to write planned-call.json instead of calling createJiraIssue MCP, with all MCP tools denied in permissions. This tests type routing, convention loading, smart defaults, field formatting, and validation without creating real Jira issues. 8 judges: 6 programmatic (output exists, valid JSON, universal defaults, correct issue type, correct project key, convention fields) and 2 LLM (description quality, convention compliance). Assisted-by: Claude:claude-opus-4-6
|
Skipping CI for Draft Pull Request. |
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: cblecker The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
Contributor
WalkthroughAdds fourteen Jira create evaluation cases covering issue types, projects, conventions, versions, custom fields, validation warnings, and credential blocking. Adds ChangesJira create evaluation coverage
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant EvaluationModel
participant JiraCreateSkill
participant Workspace
participant EvalJudges
EvaluationModel->>JiraCreateSkill: generate createJiraIssue arguments in DRY RUN
JiraCreateSkill->>Workspace: write planned-call.json and validation-results.json
Workspace->>EvalJudges: provide evaluation outputs
EvalJudges->>EvalJudges: validate structure, defaults, conventions, descriptions, and blocking
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 10✅ Passed checks (10 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 9
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/jira/evals/cases/create/case-003-epic-cntrlplane/annotations.yaml`:
- Around line 17-19: Update the create-case assertions for the CNTRLPLANE epic
to explicitly validate that additional_fields.parent.key equals
"CNTRLPLANE-500", either by adding a parent assertion or a dedicated judge
alongside the existing Epic Name check.
In
`@plugins/jira/evals/cases/create/case-004-story-hypershift-aro/annotations.yaml`:
- Around line 8-9: The annotation only validates that customfield_10855 is an
array and does not verify the requested version. Add a normalized openshift-4.22
expectation to this case’s annotations, or update the relevant judge logic to
resolve and compare the version value while preserving the existing array-shape
check.
In
`@plugins/jira/evals/cases/create/case-005-bug-ocpbugs-hypershift-rosa/annotations.yaml`:
- Around line 8-10: The annotations for this fixture only validate types and
presence, not the expected values. Update the executable assertions using
expected_version_field and expected_affects_version to require customfield_10855
to equal openshift-4.22 and the affects-version entry to exactly equal 4.21,
adding a machine-readable expected target-version value as needed.
In `@plugins/jira/evals/cases/create/case-009-story-gcp-hcp/annotations.yaml`:
- Around line 11-12: The convention_fields_applied judge must validate the value
of customfield_10028, not merely its presence. Update the relevant comparison
logic to read the expected value from expected_custom_fields and compare it with
the generated field using numeric normalization so equivalent numeric
representations pass while incorrect estimates fail.
In `@plugins/jira/evals/cases/create/case-012-epic-gcp-hcp/annotations.yaml`:
- Around line 17-20: Make the no-story-points requirement machine-checkable in
the epic evaluation annotations: add an explicit assertion or forbidden-field
check alongside the existing template and custom-field checks, targeting the
story-points field so the case fails whenever an epic contains story points.
In `@plugins/jira/evals/eval-create.yaml`:
- Around line 340-385: The blocked-case evaluation prompt lacks the validation
evidence needed to assess detection and guidance. Update the prompt’s
planned-call section to also load and display validation-results.json from the
outputs, so the judge can verify the warning and guidance; retain the existing
annotations and rubric.
- Around line 41-49: The eval assertions must enforce the complete validation
output contract. Update the relevant judge logic to require both
validation-results.json and planned-call.json, validate validation-results.json
fields and types, compare summary_warnings and security_warnings against
expected_validation_warnings, and require creation_blocked to be true for
credential-detected cases while planned-call.json is exactly {}.
- Around line 160-164: The validation logic for the createJiraIssue judge must
enforce every field in the prompt contract. Update the required fields list in
the validation block to include cloudId, projectKey, issueTypeName, summary,
description, contentFormat, and additional_fields, so malformed calls cannot
pass the structural gate.
- Around line 294-321: The validation logic in the eval’s version,
affects-version, and custom-field checks only verifies presence or broad type,
not expected values. Update the relevant checks to honor
expected_affects_version, compare each expected_custom_fields entry against its
expected_val, validate version item shape and normalized values, and use
expected_version_field when selecting the version field; preserve clear errors
and details for mismatches.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 2e1539f7-8846-45bd-9ab0-d7db415158d6
📒 Files selected for processing (29)
plugins/jira/evals/cases/create/case-001-story-cntrlplane-basic/annotations.yamlplugins/jira/evals/cases/create/case-001-story-cntrlplane-basic/input.yamlplugins/jira/evals/cases/create/case-002-bug-ocpbugs-basic/annotations.yamlplugins/jira/evals/cases/create/case-002-bug-ocpbugs-basic/input.yamlplugins/jira/evals/cases/create/case-003-epic-cntrlplane/annotations.yamlplugins/jira/evals/cases/create/case-003-epic-cntrlplane/input.yamlplugins/jira/evals/cases/create/case-004-story-hypershift-aro/annotations.yamlplugins/jira/evals/cases/create/case-004-story-hypershift-aro/input.yamlplugins/jira/evals/cases/create/case-005-bug-ocpbugs-hypershift-rosa/annotations.yamlplugins/jira/evals/cases/create/case-005-bug-ocpbugs-hypershift-rosa/input.yamlplugins/jira/evals/cases/create/case-006-task-cntrlplane-plain/annotations.yamlplugins/jira/evals/cases/create/case-006-task-cntrlplane-plain/input.yamlplugins/jira/evals/cases/create/case-007-feature-cntrlplane/annotations.yamlplugins/jira/evals/cases/create/case-007-feature-cntrlplane/input.yamlplugins/jira/evals/cases/create/case-008-feature-request-rfe/annotations.yamlplugins/jira/evals/cases/create/case-008-feature-request-rfe/input.yamlplugins/jira/evals/cases/create/case-009-story-gcp-hcp/annotations.yamlplugins/jira/evals/cases/create/case-009-story-gcp-hcp/input.yamlplugins/jira/evals/cases/create/case-010-bug-ocpbugs-version-flag/annotations.yamlplugins/jira/evals/cases/create/case-010-bug-ocpbugs-version-flag/input.yamlplugins/jira/evals/cases/create/case-011-story-hypershift-rosa/annotations.yamlplugins/jira/evals/cases/create/case-011-story-hypershift-rosa/input.yamlplugins/jira/evals/cases/create/case-012-epic-gcp-hcp/annotations.yamlplugins/jira/evals/cases/create/case-012-epic-gcp-hcp/input.yamlplugins/jira/evals/cases/create/case-013-story-bad-summary/annotations.yamlplugins/jira/evals/cases/create/case-013-story-bad-summary/input.yamlplugins/jira/evals/cases/create/case-014-bug-credentials-in-content/annotations.yamlplugins/jira/evals/cases/create/case-014-bug-credentials-in-content/input.yamlplugins/jira/evals/eval-create.yaml
Comment on lines
+17
to
+19
| CNTRLPLANE epic with parent linking. Tests Epic Name custom field | ||
| (customfield_10011) which must match the summary exactly. Also tests | ||
| parent linking via additional_fields.parent.key = "CNTRLPLANE-500". |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== File context =="
git ls-files 'plugins/jira/evals/cases/create/case-003-epic-cntrlplane/annotations.yaml'
wc -l plugins/jira/evals/cases/create/case-003-epic-cntrlplane/annotations.yaml
cat -n plugins/jira/evals/cases/create/case-003-epic-cntrlplane/annotations.yaml
echo
echo "== Nearby case schemas =="
fd -a 'annotations.yaml' plugins/jira/evals/cases/create | sed -n '1,40p'
echo
echo "== Search for parent-related expectations in create cases =="
rg -n --hidden --glob 'plugins/jira/evals/cases/create/**/annotations.yaml' 'expected_parent|parent\.key|parent link|parent_link|parent:' plugins/jira/evals/cases/createRepository: openshift-eng/ai-helpers
Length of output: 3028
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Search for annotation schema consumers =="
rg -n "expected_custom_fields|expected_validation_warnings|description_template|notes|expected_parent|parent\.key|additional_fields\.parent" plugins/jira -g '!**/node_modules/**'
echo
echo "== Inspect related create cases for any parent assertion shape =="
for f in \
plugins/jira/evals/cases/create/case-001-story-cntrlplane-basic/annotations.yaml \
plugins/jira/evals/cases/create/case-012-epic-gcp-hcp/annotations.yaml \
plugins/jira/evals/cases/create/case-007-feature-cntrlplane/annotations.yaml
do
echo "--- $f ---"
cat -n "$f"
echo
doneRepository: openshift-eng/ai-helpers
Length of output: 13958
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Eval schema around documented fields =="
sed -n '90,130p' plugins/jira/evals/eval-create.yaml
echo
echo "== Evaluator logic around custom field checks =="
sed -n '300,380p' plugins/jira/evals/eval-create.yaml
echo
echo "== Any parent assertions anywhere in evals =="
rg -n "expected_parent|parent\.key|additional_fields\.parent|parent link|parent_link" plugins/jira/evalsRepository: openshift-eng/ai-helpers
Length of output: 6572
Assert the parent link in this case
plugins/jira/evals/cases/create/case-003-epic-cntrlplane/annotations.yaml:17-19 mentions CNTRLPLANE-500 parent linking, but the case only checks the Epic Name field. Add an explicit parent assertion or a dedicated judge for additional_fields.parent.key so that failure can’t slip through.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/jira/evals/cases/create/case-003-epic-cntrlplane/annotations.yaml`
around lines 17 - 19, Update the create-case assertions for the CNTRLPLANE epic
to explicitly validate that additional_fields.parent.key equals
"CNTRLPLANE-500", either by adding a parent assertion or a dedicated judge
alongside the existing Epic Name check.
Comment on lines
+8
to
+9
| expected_version_field: customfield_10855 | ||
| expected_version_format: array |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== fixture ==\n'
cat -n plugins/jira/evals/cases/create/case-004-story-hypershift-aro/annotations.yaml
printf '\n== search for expected_version_field / format ==\n'
rg -n "expected_version_field|expected_version_format|version 4\.22|4\.22" plugins/jira/evals -S
printf '\n== search for annotation/judge handling ==\n'
rg -n "expected_version_field|expected_version_format|normalized version|resolve.*version|version value|customfield_10855" -S .Repository: openshift-eng/ai-helpers
Length of output: 15256
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== eval-create version check ==\n'
sed -n '280,340p' plugins/jira/evals/eval-create.yaml
printf '\n== case input ==\n'
cat -n plugins/jira/evals/cases/create/case-004-story-hypershift-aro/input.yaml
printf '\n== cntrlplane reference ==\n'
sed -n '1,120p' plugins/jira/reference/cntrlplane.md
printf '\n== related story case ==\n'
cat -n plugins/jira/evals/cases/create/case-001-story-cntrlplane-basic/annotations.yamlRepository: openshift-eng/ai-helpers
Length of output: 7041
Assert the target version value, not just the array shape. expected_version_format: array only checks that customfield_10855 is a list, so a wrong version ID can still pass. Add the normalized openshift-4.22 expectation here, or extend the judge to resolve and compare the version value.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@plugins/jira/evals/cases/create/case-004-story-hypershift-aro/annotations.yaml`
around lines 8 - 9, The annotation only validates that customfield_10855 is an
array and does not verify the requested version. Add a normalized openshift-4.22
expectation to this case’s annotations, or update the relevant judge logic to
resolve and compare the version value while preserving the existing array-shape
check.
Comment on lines
+8
to
+10
| expected_version_field: customfield_10855 | ||
| expected_version_format: string | ||
| expected_affects_version: [{"name": "4.21"}] |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Make the version assertions value-sensitive.
This fixture’s notes describe customfield_10855 as openshift-4.22 and affects version as 4.21, but the executable annotations only require a string and a non-null versions field. The downstream judge therefore accepts incorrect versions. Add a machine-readable expected target-version value and compare both fields exactly.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@plugins/jira/evals/cases/create/case-005-bug-ocpbugs-hypershift-rosa/annotations.yaml`
around lines 8 - 10, The annotations for this fixture only validate types and
presence, not the expected values. Update the executable assertions using
expected_version_field and expected_affects_version to require customfield_10855
to equal openshift-4.22 and the affects-version entry to exactly equal 4.21,
adding a machine-readable expected target-version value as needed.
Comment on lines
+11
to
+12
| expected_custom_fields: | ||
| customfield_10028: 2.0 |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Assert the expected Story Points value, not just field presence.
expected_custom_fields declares customfield_10028: 2.0, but the downstream convention_fields_applied judge only verifies that the field exists. An estimate of 1 or 8 would pass, so this case does not actually test the documented two-point smart default. Compare the generated value with the annotation, with numeric normalization if necessary.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/jira/evals/cases/create/case-009-story-gcp-hcp/annotations.yaml`
around lines 11 - 12, The convention_fields_applied judge must validate the
value of customfield_10028, not merely its presence. Update the relevant
comparison logic to read the expected value from expected_custom_fields and
compare it with the generated field using numeric normalization so equivalent
numeric representations pass while incorrect estimates fail.
Comment on lines
+17
to
+20
| GCP project epic with GCP-HCP team conventions. Tests Epic Name custom | ||
| field (customfield_10011) matching summary, and GCP epic template (use | ||
| case/context, current/desired state, scope, story breakdown, acceptance | ||
| criteria). No story points on epics. |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== target file =="
cat -n plugins/jira/evals/cases/create/case-012-epic-gcp-hcp/annotations.yaml
echo
echo "== search for story point mentions in eval cases =="
rg -n --hidden --glob 'plugins/jira/evals/cases/**/annotations.yaml' -e 'story points|story_points|storyPoints|no story points|points' plugins/jira/evals/cases
echo
echo "== search for forbidden/absent field conventions in eval cases =="
rg -n --hidden --glob 'plugins/jira/evals/cases/**/annotations.yaml' -e 'forbidden|must not|must_not|absent|should not|should_not|not emitted|not present|missing field' plugins/jira/evals/casesRepository: openshift-eng/ai-helpers
Length of output: 1335
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== files that define or consume annotations.yaml =="
rg -n --hidden -e 'expected_custom_fields|expected_validation_warnings|description_template|notes:' plugins/jira -g '*.py' -g '*.ts' -g '*.js' -g '*.yaml' -g '*.yml'
echo
echo "== search for any schema around annotations fields =="
rg -n --hidden -e 'expected_issue_type|expected_project_key|expected_conventions|expected_labels|expected_validation_warnings|expected_creation_blocked' .
echo
echo "== search for forbidden/negative assertions in jira eval code =="
rg -n --hidden -e 'forbidden|must not|must_not|absent|not emitted|missing field|negative assertion|unexpected field|not present' plugins/jiraRepository: openshift-eng/ai-helpers
Length of output: 18422
Make the no-story-points rule machine-checkable. The note only lives in notes; add an explicit assertion or a forbidden-field check so epics fail when story points are present.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/jira/evals/cases/create/case-012-epic-gcp-hcp/annotations.yaml`
around lines 17 - 20, Make the no-story-points requirement machine-checkable in
the epic evaluation annotations: add an explicit assertion or forbidden-field
check alongside the existing template and custom-field checks, targeting the
story-points field so the case fails whenever an epic contains story points.
Comment on lines
+41
to
+49
| Also write `validation-results.json` with: | ||
| { | ||
| "summary_warnings": [...], | ||
| "security_warnings": [...], | ||
| "creation_blocked": false | ||
| } | ||
|
|
||
| If security validation detects credentials, set creation_blocked to | ||
| true and leave planned-call.json empty (write {} only). |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Assert the complete validation output contract.
The configuration requires both output files, but no judge requires validation-results.json or compares its warnings with expected_validation_warnings. Additionally, blocked cases pass this judge with any parseable JSON, not specifically {}. Require both files, validate their schemas, compare expected warnings, and require creation_blocked: true for blocked cases.
Also applies to: 109-117, 126-164
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/jira/evals/eval-create.yaml` around lines 41 - 49, The eval
assertions must enforce the complete validation output contract. Update the
relevant judge logic to require both validation-results.json and
planned-call.json, validate validation-results.json fields and types, compare
summary_warnings and security_warnings against expected_validation_warnings, and
require creation_blocked to be true for credential-detected cases while
planned-call.json is exactly {}.
Comment on lines
+160
to
+164
| required = ["projectKey", "issueTypeName", "summary", "contentFormat"] | ||
| missing = [f for f in required if f not in data] | ||
| if missing: | ||
| return (False, f"Missing required fields: {', '.join(missing)}") | ||
| return (True, "Valid JSON with all required fields") |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Validate all required createJiraIssue fields.
The prompt’s contract includes cloudId, description, and additional_fields, but this judge only requires four fields. A malformed MCP call can therefore pass the structural gate.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/jira/evals/eval-create.yaml` around lines 160 - 164, The validation
logic for the createJiraIssue judge must enforce every field in the prompt
contract. Update the required fields list in the validation block to include
cloudId, projectKey, issueTypeName, summary, description, contentFormat, and
additional_fields, so malformed calls cannot pass the structural gate.
Comment on lines
+294
to
+321
| # Check version format | ||
| exp_vf = ann.get("expected_version_format") | ||
| if exp_vf: | ||
| tv = af.get("customfield_10855", data.get("customfield_10855")) | ||
| if tv is None: | ||
| errors.append("customfield_10855 (Target Version) not set") | ||
| elif exp_vf == "array" and not isinstance(tv, list): | ||
| errors.append(f"Target Version should be array, got {type(tv).__name__}: {tv}") | ||
| elif exp_vf == "string" and not isinstance(tv, str): | ||
| errors.append(f"Target Version should be string, got {type(tv).__name__}: {tv}") | ||
| else: | ||
| details.append(f"version_format={exp_vf}") | ||
| # Check affects version (OCPBUGS bugs) | ||
| exp_av = ann.get("expected_affects_version") | ||
| if exp_av: | ||
| vers = af.get("versions", data.get("versions")) | ||
| if vers is None: | ||
| errors.append("versions (Affects Version) not set") | ||
| else: | ||
| details.append(f"affects_version set") | ||
| # Check custom fields | ||
| exp_cf = ann.get("expected_custom_fields", {}) | ||
| for field_id, expected_val in exp_cf.items(): | ||
| actual_val = af.get(field_id, data.get(field_id)) | ||
| if actual_val is None: | ||
| errors.append(f"Custom field {field_id} not set") | ||
| else: | ||
| details.append(f"{field_id} set") |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Compare convention values, not only their presence or type.
expected_affects_version is only checked for non-null, expected_custom_fields ignores expected_val, and version validation accepts any list/string without checking item shape, normalization, or expected_version_field. Incorrect Jira fields can pass the structural judge.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/jira/evals/eval-create.yaml` around lines 294 - 321, The validation
logic in the eval’s version, affects-version, and custom-field checks only
verifies presence or broad type, not expected values. Update the relevant checks
to honor expected_affects_version, compare each expected_custom_fields entry
against its expected_val, validate version item shape and normalized values, and
use expected_version_field when selecting the version field; preserve clear
errors and details for mismatches.
Comment on lines
+340
to
+385
| prompt: | | ||
| You are evaluating whether a Jira issue description follows the correct | ||
| type-specific template from the jira:create skill. | ||
|
|
||
| The planned createJiraIssue call: | ||
|
|
||
| {% for path, content in outputs.files.items() if path.endswith('planned-call.json') %} | ||
| {{ content }} | ||
| {% endfor %} | ||
|
|
||
| Expected outcomes for this test case: | ||
|
|
||
| {{ annotations }} | ||
|
|
||
| The 'description_template' field indicates which template the description | ||
| should follow. Key template requirements: | ||
|
|
||
| - **story**: User story format (As a / I want / so that), acceptance criteria | ||
| - **bug**: Problem description, version, reproducibility, steps to reproduce, | ||
| actual results, expected results | ||
| - **epic**: Objective, scope, acceptance criteria (3-6 outcomes), Epic Name field | ||
| - **feature**: Market problem, strategic value, success criteria | ||
| - **task**: What/why, acceptance criteria, technical details | ||
| - **feature-request**: Title, nature/description, business requirements | ||
| - **gcp-story**: GCP team template — user story format, context/background, | ||
| requirements, technical approach, dependencies, acceptance criteria checklist | ||
| - **gcp-epic**: GCP template — use case/context, current/desired state, scope, | ||
| story breakdown, acceptance criteria | ||
|
|
||
| Evaluate on a 1-5 scale: | ||
|
|
||
| Score 1: No description, or description is a single line with no structure. | ||
| Score 2: Some structure but wrong template — e.g., a bug using story format, | ||
| or missing most required sections for the correct template. | ||
| Score 3: Correct template identified and partially followed — has some required | ||
| sections but is missing important ones (e.g., story without AC, bug | ||
| without steps to reproduce). | ||
| Score 4: Correct template fully followed with all required sections present. | ||
| Content is specific to the summary/context provided. | ||
| Score 5: Excellent — correct template, all sections present with specific and | ||
| actionable content, well-formatted Markdown. | ||
|
|
||
| If creation_blocked is true in annotations, evaluate only whether the skill | ||
| correctly identified the security issue — score 4 for correct detection, | ||
| 5 for detection with helpful guidance about what was found. | ||
|
|
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Provide validation evidence to the blocked-case LLM judge.
For blocked cases, the rubric scores detection and helpful guidance, but the prompt supplies only planned-call.json and annotations—not validation-results.json, where the warning and guidance are written. Include the validation output or make this assertion deterministic.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/jira/evals/eval-create.yaml` around lines 340 - 385, The blocked-case
evaluation prompt lacks the validation evidence needed to assess detection and
guidance. Update the prompt’s planned-call section to also load and display
validation-results.json from the outputs, so the judge can verify the warning
and guidance; retain the existing annotations and rubric.
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
jira:createskill (which implicitly testsjira:jira-conventions)planned-call.jsoninstead of callingcreateJiraIssueMCP, with all MCP tools denied — no Jira auth needed, no garbage issues createdTest case coverage
Test plan
make lint— passes with A+ grade (verified locally)claude --plugin-dir /tmp/agent-eval-harness --plugin-dir plugins/jira -p "/eval-run --config plugins/jira/evals/eval-create.yaml --model claude-sonnet-4-20250514 --cases case-001-story-cntrlplane-basic"Summary by CodeRabbit