OTA-2024, OTA-2070: Add eval test cases for cluster-update skills - #34
Open
jhadvig wants to merge 1 commit into
Open
OTA-2024, OTA-2070: Add eval test cases for cluster-update skills#34jhadvig wants to merge 1 commit into
jhadvig wants to merge 1 commit into
Conversation
openshift-ci-robot
added
the
jira/valid-reference
|
@jhadvig: This pull request references OTA-2024 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target either version "5.0." or "openshift-5.0.", but it targets "openshift-5.1" instead. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: jhadvig The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Member
|
evals manual execution against Expand. Execution logs |
Member
|
evals manual test run for Expand. Execution logs |
3 tasks
Member
Author
Eval results analysis (from @jrangelramos runs)update-advisor: 5/5 passed 👍 product-lifecycle: 4/7 passed, 3 failures:
Plan:
|
Member
|
/verified |
|
@jrangelramos: The DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
Member
|
@jrangelramos what's the status here? |
Member
|
/verified by Jefferson Ramos |
openshift-ci-robot
added
the
verified
|
@jrangelramos: This PR has been marked as verified by DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
Member
Member
Author
|
ccing @harche |
Member
|
Update Advisory evals tests with OpenAI 🔍 Update Advisory run.sh Logs (openai / gpt-5.4)🔍 Update Advisory (sandbox container logs) |
Member
|
Product Lifecycle evals tests with OpenAI 🔍 run.sh Logs (openai / gpt-5.4)🔍 sandbox container logs |
harche
reviewed
Jul 21, 2026
harche
left a comment
harche
left a comment
Contributor
There was a problem hiding this comment.
Thanks for salvaging these evals and getting cluster-update coverage started! Left a few questions inline, mostly around the product-lifecycle ground truth.
| expected: | ||
| olm_check_ran: true | ||
| operators_checked: 5 | ||
| operators_without_lifecycle_data: 2 |
Contributor
There was a problem hiding this comment.
Should this be 1? Looking at the live API, 4 of the 5 packages have entries with a matching package field (cluster-logging, web-terminal, compliance-operator, openshift-pipelines-operator-rh), only devworkspace-operator is missing. The PR description says 3 and the yaml says 2, so I wanted to double check which count is intended.
Contributor
There was a problem hiding this comment.
Looks like the eval run above confirms this one. In the gpt-5.4 run this is the single failing case:
AssertionError: plc_proposal_olm_batch_check: field 'operators_without_lifecycle_data' expected 2, got 3
So we now have three different answers depending on lookup path: 1 when matching the package field across the full product list (what olm-check in plc_lookup.py does, only devworkspace-operator is missing), 2 as expected here, and 3 from the agent in that run, consistent with name searches that miss openshift-pipelines-operator-rh (a name query for "pipelines" returns nothing even though the package exists). Would it make sense to pin the expectation to the olm-check package match, or word the query so the agent has to use it?
Member
Author
There was a problem hiding this comment.
You're right that the count is off. This ties into PR #41 too, which adds the nuance that lifecycle data should only count if the API has an entry matching the installed version, not just any version of the package. @jrangelramos can you run olm-check against these 5 operators and confirm what the actual count should be? That'll settle whether it's 1, 2, or something else depending on version matching.
Member
Author
There was a problem hiding this comment.
The three-way mismatch is a great catch. Agree we should pin the expectation to what olm-check actually returns.... the eval should test the same lookup path the agent uses. @jrangelramos would it make sense to also tighten the query wording so the agent is directed to use the olm-check command rather than ad-hoc name searches?
Member
There was a problem hiding this comment.
This is what PR #41 is addressing. What I observed on logs is it attempts olm-check and if not found attempt by product name. On most runs I tried (with claude and gpt) it returned count 2 for operators_with_lifecycle_data. So in order to exercise the olm-check part only of the skill, (which is what this test case is about) I agree we should tghten the query wording to be more explicit on this. and have expect count as 3
Member
Author
There was a problem hiding this comment.
@jrangelramos just want to make sure I'm reading this right. You say tighten the query to exercise olm-check only, and set the expected count to 3. But olm-check does a package-field match across the full product list, which finds 4 of 5 (only devworkspace-operator is missing), so operators_without_lifecycle_data would be 1 not 3. The count of 3 matches the name-search path that misses openshift-pipelines-operator-rh and compliance-operator too.... which is the opposite of olm-check-only.
Did you mean the expected should be 1 (olm-check finds 4, misses 1)? Or are you factoring in version matching from PR #41 where having an API entry doesn't count unless the installed version matches?
Member
There was a problem hiding this comment.
Hey @jhadvig Sorry for the confusion. Based on the what the Live API returns (without filtering), I would expect 2 operators without Lifecycle data, crossing operator against given OCP 4.21 version. Here is why:
Verification
So for the below operator/packages against ocp 4.21:
- compliance-operator v1.9.0
- cluster-logging v6.5.1
- devworkspace-operator v0.41.0
- openshift-pipelines-operator-rh v1.22.0
- web-terminal v1.16.0
Here is what the Live API has and the given products
#===========================================
# Retrieve ALL products (no query param ?name=Openshift)
#===========================================
curl -s -H "User-Agent: plc-lookup/1.0" 'https://access.redhat.com/product-life-cycles/api/v2/products' > products.json
# Returns all 237 products, not just the ~14 when passing name=Openshift
#===========================================
# OCP 4.21 - compliance-operator v1.9.0
#===========================================
cat products.json | jq '[.data[] | select(.package == "compliance-operator") | {product: .name, package, versions: [.versions[] | select(.openshift_compatibility != null and (.openshift_compatibility | contains("4.21")) and (.name | contains("1.9"))) | {version: .name, status: .type, ocp_versions: .openshift_compatibility}]}]'
[
{
"product": "compliance operator",
"package": "compliance-operator",
"versions": [
{
"version": "1.9",
"status": "Full Support",
"ocp_versions": "4.12, 4.14, 4.16, 4.18, 4.19, 4.20, 4.21"
}
]
}
]
#===========================================
# OCP 4.21 - cluster-logging v6.5.1
#===========================================
cat products.json | jq '[.data[] | select(.package == "cluster-logging") | {product: .name, package, versions: [.versions[] | select(.openshift_compatibility != null and (.openshift_compatibility | contains("4.21")) and (.name | contains("6.5"))) | {version: .name, status: .type, ocp_versions: .openshift_compatibility}]}]'
[
{
"product": "logging for Red Hat OpenShift",
"package": "cluster-logging",
"versions": [
{
"version": "6.5",
"status": "Full Support",
"ocp_versions": "4.19, 4.20, 4.21, 4.22"
}
]
}
]
#===========================================
# OCP 4.21 - devworkspace-operator v0.41.0
#===========================================
cat products.json | jq '[.data[] | select(.package == "devworkspace-operator") | {product: .name, package, versions: [.versions[] | select(.openshift_compatibility != null and (.openshift_compatibility | contains("4.21")) and (.name | contains("0.41"))) | {version: .name, status: .type, ocp_versions: .openshift_compatibility}]}]'
[]
#===========================================
# OCP 4.21 - openshift-pipelines-operator-rh v1.22.0
#===========================================
cat products.json | jq '[.data[] | select(.package == "openshift-pipelines-operator-rh") | {product: .name, package, versions: [.versions[] | select(.openshift_compatibility != null and (.openshift_compatibility | contains("4.21")) and (.name | contains("1.22"))) | {version: .name, status: .type, ocp_versions: .openshift_compatibility}]}]'
[
{
"product": "Red Hat OpenShift Pipelines",
"package": "openshift-pipelines-operator-rh",
"versions": [
{
"version": "1.22",
"status": "Full Support",
"ocp_versions": "4.16, 4.18, 4.19, 4.20, 4.21, 4.22"
}
]
}
]
#===========================================
# OCP 4.21 - web-terminal v1.16.0
#===========================================
cat products.json | jq '[.data[] | select(.package == "web-terminal") | {product: .name, package, versions: [.versions[] | select(.openshift_compatibility != null and (.openshift_compatibility | contains("4.21")) and (.name | contains("1.16"))) | {version: .name, status: .type, ocp_versions: .openshift_compatibility}]}]'
[
{
"product": "web terminal operator",
"package": "web-terminal",
"versions": []
}
]Conclusion
for Target: OCP 4.21
| Package | Version | Package Found | Status | Compatibility |
|---|---|---|---|---|
| compliance-operator | v1.9.0 | Yes | Full Support | OCP 4.21 |
| cluster-logging | v6.5.1 | Yes | Full Support | OCP 4.21 |
| devworkspace-operator | v0.41.0 | No | - | - |
| openshift-pipelines-operator-rh | v1.22.0 | Yes | Full Support | OCP 4.21 |
| web-terminal | v1.16.0 | Yes | Version not found | - |
So I would expect 2 operators without lifecycle data (devworkspace-operator, web-terminal).
possible approach
I think for the olm-check batch check we could remove name=Openshift, so the API will return all products, to be iterated by the cli, no fallback (replace hypen by space) is needed in this case.
Member
There was a problem hiding this comment.
the skill script plc_lookup.py has evolved (changed by #43) to now return a better veredict on the expected results. So that I modified the test logic to now compare the result from the cli (plc_lookup.py) counting "not found" (or error) as not tracked, version tracked whenever found (since version is specified on every operator test) and ocp compatbible according to the api result.
| "\($prod) \(.name) - \(.type)"' | ||
|
|
||
| # Batch check — find operators by package name in the "OpenShift" product set | ||
| curl -s "https://access.redhat.com/product-life-cycles/api/v1/products?name=OpenShift" \ |
Contributor
There was a problem hiding this comment.
Could this query undercount? name=OpenShift only returns products with OpenShift in their name, while olm-check in plc_lookup.py matches package across the full product list, which also finds compliance-operator and openshift-pipelines-operator-rh. Maybe that explains the mismatch with the expected count in test 1?
Member
Author
There was a problem hiding this comment.
Yeah this is the same root cause as the count issue above, the curl approach takes a different lookup path than olm-check. Will fix the README alongside the ground truth update.
Member
There was a problem hiding this comment.
README changed with batch check restricted to search by package only matching new param format.
|
|
||
| ## Ground truth | ||
|
|
||
| Expected values come from the live PLC API. Verify with: |
Contributor
There was a problem hiding this comment.
The PR description mentions the CLI not being on main yet, but plc_lookup.py is already there and SKILL.md routes all queries through it. Would it make sense to derive ground truth with the CLI the agent will actually run? The curl commands are handy but take a different lookup path than olm-check.
Member
Author
There was a problem hiding this comment.
Agree, now that the CLI is on main we should use it for ground truth. Will update the README once we settle the count question above.
Member
|
|
||
| # Test 4: compliance-operator — found by product name search, v1.9 is supported. | ||
| # Note: searching by package name may not find this operator, but searching | ||
| # by product name ("compliance operator") does. The agent may use either approach. |
Contributor
There was a problem hiding this comment.
Is this note still accurate? Package search does seem to find this one now, the API returns "package": "compliance-operator" for this query.
Member
Author
There was a problem hiding this comment.
Yeah youre right, the API does index it by package now. Updated the comment in 41e61b6 to reflect that both lookup paths work.
| required: ["product_found", "has_supported_version_for_421"] | ||
| expected: | ||
| product_found: true | ||
| has_supported_version_for_421: false |
Contributor
There was a problem hiding this comment.
This holds today (latest listed web-terminal version is 1.15, compatible with 4.20 only), but the scenario has v1.16 installed, so once the API adds a 1.16 entry with 4.21 compatibility this expectation flips. Same concern for the OCP 4.21 and 4.14 status tests below as support phases move. Would the _fn verify.py pattern from the framework README be a better fit for values that come from a live API?
Member
Author
There was a problem hiding this comment.
Good point about the fragility. The _fn verify.py pattern is the right fix for these live-API-dependent expectations. Id rather do that as a follow-up PR though, it's a bigger change and shouldn't block this one. @jrangelramos what do you think?
Member
There was a problem hiding this comment.
yea I think it may fit well, the custom function mappend on _fn could make use of the API in advance to properly assert whether the product is compatible or not with the version.
Member
There was a problem hiding this comment.
This is now addressed with a suggested approach. The verify.py has been implemented. Each individual test now uses _fn since lifecycle data (support status) may change along the time.
| status: "extended" | ||
|
|
||
| # Test 7: Multiple operators batch check from proposal context. | ||
| # Only test the 2 operators that HAVE lifecycle data (cluster-logging, web-terminal). |
Contributor
There was a problem hiding this comment.
Small note, per the API at least 4 of the 5 operators have lifecycle data now, so this comment may be stale. The expectations themselves still look right though.
Member
Author
There was a problem hiding this comment.
Updated the wording in 41e61b6, the "only" was misleading.
| target OCP version 4.21 is still in full support. | ||
|
|
||
| Use the product-lifecycle skill to look up "Red Hat OpenShift | ||
| Container Platform" and check version 4.21's. Report whether |
Contributor
There was a problem hiding this comment.
Tiny nit, check version 4.21's. reads like a word is missing.
Member
Author
There was a problem hiding this comment.
Fixed in 41e61b6 - reads "check version 4.21's lifecycle status" now.
| expected: | ||
| decision: "block" | ||
| etcd_issue_detected: true | ||
| unhealthy_member: "etcd-master-2" |
Contributor
There was a problem hiding this comment.
Since unhealthy_member is not in the schema's required list, an agent that omits it passes schema validation but fails here, the static matcher asserts every expected field is present (test_eval.py#L141-L145). Also evals/CLAUDE.md suggests avoiding free text fields in schemas. Would it be safer to either add it to required or drop it from expected?
Member
Author
There was a problem hiding this comment.
Good catch 👍 Dropped unhealthy_member from expected in 41e61b6. The eval still validates decision and etcd_issue_detected which covers the important bits, and avoids the free-text field footgun you flagged.
| update-advisor skill's decision framework to classify findings as | ||
| blockers, warnings, or informational. | ||
|
|
||
| Decision matrix: |
Contributor
There was a problem hiding this comment.
With the full matrix embedded in the system prompt, does the eval still test that the agent discovers and applies the SKILL.md decision framework? A shorter prompt that just points at the skill might be a stronger test, curious what you think.
Member
Author
There was a problem hiding this comment.
Fair question. Right now the eval tests whether the agent can apply the framework correctly given the data, not whether it discovers the framework. A stripped-down prompt would be a stronger end-to-end test but might also make the eval flakier since the agent has to find SKILL.md and extract the rules. @jrangelramos do you have a preference here? We could start with the explicit prompt for reliability and add a "discovery" variant later.
Member
There was a problem hiding this comment.
Good point. I agree with @harche. I would go for stripping the decision matrix out of the system_prompt. The decision matrix is already on the SKILL.md and that would be closer to what we have real cvo agentic runs
Member
There was a problem hiding this comment.
This one is now addressed as well. 👍 I removed the duplication from the system_prompt, thus leaving the decision matrix on the SKILL only.
openshift-ci-robot
removed
the
verified
3 tasks
Contributor
|
/override ci/prow/eval |
|
@harche: Overrode contexts on behalf of harche: ci/prow/eval DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Member
|
Product Lifecycle and Update Advisor new batch of tests - claude opus4-6 🔍 Expand: Eval Run Logs🔍 Expand: Sanbox Container Logs |
Member
|
Product Lifecycle and Update Advisor new batch of tests - openai gpt-5.4 🔍 Expand: Eval Run Logs🔍 Expand: Sandbox Container Logs |
Member
|
/verified by Jefferson Ramos |
openshift-ci-robot
added
the
verified
|
@jrangelramos: This PR has been marked as verified by DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
jrangelramos
force-pushed
the
jhadvig/ota-2024-cluster-update-evals
branch
from
July 30, 2026 17:57
d3d1190 to
6da4fd2
Compare
openshift-ci-robot
removed
the
verified
Add eval coverage for update-advisor (5 test cases) and product-lifecycle (7 test cases). Cherry-picked from closed PR openshift#13 with fixes applied: - Fix decision enum: caution → warn to match SKILL.md - Fix PLCC acronym → "Product Life Cycle API" in system prompts - Make product-lifecycle test cases tool-agnostic (no plc_lookup.py references since the CLI doesn't exist on main yet) - Rewrite README ground truth commands using curl/jq No skill code changes — eval-only PR. Co-Authored-By: Jakub Hadvig <jhadvig@redhat.com> Co-Authored-By: Jefferson Ramos <jeramos@redhat.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
jrangelramos
force-pushed
the
jhadvig/ota-2024-cluster-update-evals
branch
from
July 30, 2026 18:03
6da4fd2 to
bc7706d
Compare
Contributor
|
/override ci/prow/eval |
|
@harche: Overrode contexts on behalf of harche: ci/prow/eval DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@jhadvig: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Member
|
Test results after bc7706d recent changes model openai gpt-5.4 🔍 Expand: Eval Run Logs🔍 Expand: Sandbox Container Logs |
Member
|
/verified by Jefferson Ramos |
openshift-ci-robot
added
the
verified
|
@jrangelramos: This PR has been marked as verified by DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
Member
openshift-ci
Bot
changed the title
|
@jhadvig: This pull request references OTA-2024 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target either version "5.0." or "openshift-5.0.", but it targets "openshift-5.1" instead. This pull request references OTA-2070 which is a valid jira issue. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
Member
|
/retitle OTA-2024, OTA-2070: Add eval test cases for cluster-update skills |
openshift-ci
Bot
changed the title
Member
|
/lgtm |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
What changed
Salvaged eval files from #13 (closed) with these fixes applied:
caution→warnto match the decision matrix inupdate-advisor/SKILL.mdplc_lookup.pyreferences from schema descriptions since the CLI doesn't exist on main yet (PR B scope)plc_lookup.pyTest scenarios
update-advisor (5 cases covering the full decision matrix):
advisor_healthy_cluster_recommendrecommendadvisor_degraded_operator_warnwarnadvisor_api_deprecation_blockblockadvisor_etcd_unhealthy_blockblockadvisor_errored_checks_escalateescalateproduct-lifecycle (7 cases covering PLC API queries):
plc_proposal_olm_batch_checkplc_cluster_logging_supportedplc_web_terminal_compat_checkplc_compliance_operator_statusplc_ocp_platform_statusplc_ocp_old_version_extendedplc_batch_known_operators_onlyTest plan
python3 -m pytest evals/ --collect-only -k "update-advisor or product-lifecycle"shows 12 casesbash evals/run.sh -k "update-advisor"bash evals/run.sh -k "product-lifecycle"ls evals/workspace/skills/{update-advisor,product-lifecycle}/SKILL.md/cc @wking
🤖 Generated with Claude Code