OTA-2070: Return per-operator verdicts from olm-check - #43
Merged
openshift-merge-bot[bot] merged 1 commit intoJul 29, 2026
Merged
Conversation
openshift-ci-robot
added
the
jira/valid-reference
|
@jrangelramos: This pull request references OTA-2070 which is a valid jira issue. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
jrangelramos
force-pushed
the
fix/olm-check-per-operator-verdicts
branch
from
July 24, 2026 02:59
d316922 to
922f52f
Compare
1 task
Member
Author
|
Test logs using combined evals from jhadvig#2 on top of #34. Run with claude as provider 🔍 Expand: Eval Run Logs$ CLAUDE_CODE_USE_VERTEX=1 \
ANTHROPIC_VERTEX_PROJECT_ID=itpc-gcp-hcm-pe-eng-claude \
ANTHROPIC_MODEL=claude-opus-4-6 \
CLOUD_ML_REGION=us-east5 \
EVAL_HEALTH_TIMEOUT=180 \
bash evals/run.sh -k "product-lifecycle" --timeout=600
Starting provider containers...
claude: port 18080 (container d77ffafd9aca)
Waiting for servers...
claude: ready
Running evals...
/usr/lib/python3.14/site-packages/pytest_asyncio/plugin.py:211: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset.
The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session"
warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET))
=============================================================================================== test session starts ================================================================================================
platform linux -- Python 3.14.3, pytest-8.3.5, pluggy-1.6.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/jeramos/pixaa/test/test-ocp5/agentic-skills/evals
configfile: pytest.ini
plugins: anyio-4.12.1, asyncio-1.1.0, xdist-3.7.0, timeout-2.4.0
asyncio: mode=Mode.AUTO, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
timeout: 600.0s
timeout method: signal
timeout func_only: False
collected 228 items / 186 deselected / 42 selected
evals/skills/test_eval.py::test_skill[claude-product-lifecycle-plc_proposal_olm_batch_check] PASSED [ 2%]
evals/skills/test_eval.py::test_skill[claude-product-lifecycle-plc_cluster_logging_supported] PASSED [ 4%]
evals/skills/test_eval.py::test_skill[claude-product-lifecycle-plc_web_terminal_compat_check] PASSED [ 7%]
evals/skills/test_eval.py::test_skill[claude-product-lifecycle-plc_compliance_operator_status] PASSED [ 9%]
evals/skills/test_eval.py::test_skill[claude-product-lifecycle-plc_ocp_platform_status] PASSED [ 11%]
evals/skills/test_eval.py::test_skill[claude-product-lifecycle-plc_ocp_old_version_extended] PASSED [ 14%]
evals/skills/test_eval.py::test_skill[claude-product-lifecycle-plc_batch_known_operators_only] PASSED [ 16%]
evals/skills/test_eval.py::test_skill[gemini-product-lifecycle-plc_proposal_olm_batch_check] SKIPPED (No server for gemini) [ 19%]
evals/skills/test_eval.py::test_skill[gemini-product-lifecycle-plc_cluster_logging_supported] SKIPPED (No server for gemini) [ 21%]
evals/skills/test_eval.py::test_skill[gemini-product-lifecycle-plc_web_terminal_compat_check] SKIPPED (No server for gemini) [ 23%]
evals/skills/test_eval.py::test_skill[gemini-product-lifecycle-plc_compliance_operator_status] SKIPPED (No server for gemini) [ 26%]
evals/skills/test_eval.py::test_skill[gemini-product-lifecycle-plc_ocp_platform_status] SKIPPED (No server for gemini) [ 28%]
evals/skills/test_eval.py::test_skill[gemini-product-lifecycle-plc_ocp_old_version_extended] SKIPPED (No server for gemini) [ 30%]
evals/skills/test_eval.py::test_skill[gemini-product-lifecycle-plc_batch_known_operators_only] SKIPPED (No server for gemini) [ 33%]
evals/skills/test_eval.py::test_skill[openai-product-lifecycle-plc_proposal_olm_batch_check] SKIPPED (No server for openai) [ 35%]
evals/skills/test_eval.py::test_skill[openai-product-lifecycle-plc_cluster_logging_supported] SKIPPED (No server for openai) [ 38%]
evals/skills/test_eval.py::test_skill[openai-product-lifecycle-plc_web_terminal_compat_check] SKIPPED (No server for openai) [ 40%]
evals/skills/test_eval.py::test_skill[openai-product-lifecycle-plc_compliance_operator_status] SKIPPED (No server for openai) [ 42%]
evals/skills/test_eval.py::test_skill[openai-product-lifecycle-plc_ocp_platform_status] SKIPPED (No server for openai) [ 45%]
evals/skills/test_eval.py::test_skill[openai-product-lifecycle-plc_ocp_old_version_extended] SKIPPED (No server for openai) [ 47%]
evals/skills/test_eval.py::test_skill[openai-product-lifecycle-plc_batch_known_operators_only] SKIPPED (No server for openai) [ 50%]
evals/skills/test_eval.py::test_skill[deepagents-claude-product-lifecycle-plc_proposal_olm_batch_check] SKIPPED (No server for deepagents-claude) [ 52%]
evals/skills/test_eval.py::test_skill[deepagents-claude-product-lifecycle-plc_cluster_logging_supported] SKIPPED (No server for deepagents-claude) [ 54%]
evals/skills/test_eval.py::test_skill[deepagents-claude-product-lifecycle-plc_web_terminal_compat_check] SKIPPED (No server for deepagents-claude) [ 57%]
evals/skills/test_eval.py::test_skill[deepagents-claude-product-lifecycle-plc_compliance_operator_status] SKIPPED (No server for deepagents-claude) [ 59%]
evals/skills/test_eval.py::test_skill[deepagents-claude-product-lifecycle-plc_ocp_platform_status] SKIPPED (No server for deepagents-claude) [ 61%]
evals/skills/test_eval.py::test_skill[deepagents-claude-product-lifecycle-plc_ocp_old_version_extended] SKIPPED (No server for deepagents-claude) [ 64%]
evals/skills/test_eval.py::test_skill[deepagents-claude-product-lifecycle-plc_batch_known_operators_only] SKIPPED (No server for deepagents-claude) [ 66%]
evals/skills/test_eval.py::test_skill[deepagents-gemini-product-lifecycle-plc_proposal_olm_batch_check] SKIPPED (No server for deepagents-gemini) [ 69%]
evals/skills/test_eval.py::test_skill[deepagents-gemini-product-lifecycle-plc_cluster_logging_supported] SKIPPED (No server for deepagents-gemini) [ 71%]
evals/skills/test_eval.py::test_skill[deepagents-gemini-product-lifecycle-plc_web_terminal_compat_check] SKIPPED (No server for deepagents-gemini) [ 73%]
evals/skills/test_eval.py::test_skill[deepagents-gemini-product-lifecycle-plc_compliance_operator_status] SKIPPED (No server for deepagents-gemini) [ 76%]
evals/skills/test_eval.py::test_skill[deepagents-gemini-product-lifecycle-plc_ocp_platform_status] SKIPPED (No server for deepagents-gemini) [ 78%]
evals/skills/test_eval.py::test_skill[deepagents-gemini-product-lifecycle-plc_ocp_old_version_extended] SKIPPED (No server for deepagents-gemini) [ 80%]
evals/skills/test_eval.py::test_skill[deepagents-gemini-product-lifecycle-plc_batch_known_operators_only] SKIPPED (No server for deepagents-gemini) [ 83%]
evals/skills/test_eval.py::test_skill[deepagents-openai-product-lifecycle-plc_proposal_olm_batch_check] SKIPPED (No server for deepagents-openai) [ 85%]
evals/skills/test_eval.py::test_skill[deepagents-openai-product-lifecycle-plc_cluster_logging_supported] SKIPPED (No server for deepagents-openai) [ 88%]
evals/skills/test_eval.py::test_skill[deepagents-openai-product-lifecycle-plc_web_terminal_compat_check] SKIPPED (No server for deepagents-openai) [ 90%]
evals/skills/test_eval.py::test_skill[deepagents-openai-product-lifecycle-plc_compliance_operator_status] SKIPPED (No server for deepagents-openai) [ 92%]
evals/skills/test_eval.py::test_skill[deepagents-openai-product-lifecycle-plc_ocp_platform_status] SKIPPED (No server for deepagents-openai) [ 95%]
evals/skills/test_eval.py::test_skill[deepagents-openai-product-lifecycle-plc_ocp_old_version_extended] SKIPPED (No server for deepagents-openai) [ 97%]
evals/skills/test_eval.py::test_skill[deepagents-openai-product-lifecycle-plc_batch_known_operators_only] SKIPPED (No server for deepagents-openai) [100%]
============================================================================ 7 passed, 35 skipped, 186 deselected in 719.06s (0:11:59) =============================================================================
eval-claude🔍 Expand: Container Logs |
Member
Author
|
Test logs using combined evals from jhadvig#2 on top of #34. Run with openai as provider 🔍 Expand: Eval Run Logs$ OPENAI_API_KEY=${OPENAI_API_KEY} \
OPENAI_MODEL=gpt-5.4 \
EVAL_PROVIDERS=openai \
EVAL_HEALTH_TIMEOUT=180 \
CLAUDE_CODE_USE_VERTEX= \
ANTHROPIC_VERTEX_PROJECT_ID= \
GOOGLE_APPLICATION_CREDENTIALS= \
bash evals/run.sh -k "product-lifecycle" --timeout=600
Starting provider containers...
openai: port 18080 (container 5d95caaddb17)
Waiting for servers...
openai: ready
Running evals...
/usr/lib/python3.14/site-packages/pytest_asyncio/plugin.py:211: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset.
The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session"
warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET))
=============================================================================================== test session starts ================================================================================================
platform linux -- Python 3.14.3, pytest-8.3.5, pluggy-1.6.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/jeramos/pixaa/test/test-ocp5/agentic-skills/evals
configfile: pytest.ini
plugins: anyio-4.12.1, asyncio-1.1.0, xdist-3.7.0, timeout-2.4.0
asyncio: mode=Mode.AUTO, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
timeout: 600.0s
timeout method: signal
timeout func_only: False
collected 228 items / 186 deselected / 42 selected
evals/skills/test_eval.py::test_skill[claude-product-lifecycle-plc_proposal_olm_batch_check] SKIPPED (No server for claude) [ 2%]
evals/skills/test_eval.py::test_skill[claude-product-lifecycle-plc_cluster_logging_supported] SKIPPED (No server for claude) [ 4%]
evals/skills/test_eval.py::test_skill[claude-product-lifecycle-plc_web_terminal_compat_check] SKIPPED (No server for claude) [ 7%]
evals/skills/test_eval.py::test_skill[claude-product-lifecycle-plc_compliance_operator_status] SKIPPED (No server for claude) [ 9%]
evals/skills/test_eval.py::test_skill[claude-product-lifecycle-plc_ocp_platform_status] SKIPPED (No server for claude) [ 11%]
evals/skills/test_eval.py::test_skill[claude-product-lifecycle-plc_ocp_old_version_extended] SKIPPED (No server for claude) [ 14%]
evals/skills/test_eval.py::test_skill[claude-product-lifecycle-plc_batch_known_operators_only] SKIPPED (No server for claude) [ 16%]
evals/skills/test_eval.py::test_skill[gemini-product-lifecycle-plc_proposal_olm_batch_check] SKIPPED (No server for gemini) [ 19%]
evals/skills/test_eval.py::test_skill[gemini-product-lifecycle-plc_cluster_logging_supported] SKIPPED (No server for gemini) [ 21%]
evals/skills/test_eval.py::test_skill[gemini-product-lifecycle-plc_web_terminal_compat_check] SKIPPED (No server for gemini) [ 23%]
evals/skills/test_eval.py::test_skill[gemini-product-lifecycle-plc_compliance_operator_status] SKIPPED (No server for gemini) [ 26%]
evals/skills/test_eval.py::test_skill[gemini-product-lifecycle-plc_ocp_platform_status] SKIPPED (No server for gemini) [ 28%]
evals/skills/test_eval.py::test_skill[gemini-product-lifecycle-plc_ocp_old_version_extended] SKIPPED (No server for gemini) [ 30%]
evals/skills/test_eval.py::test_skill[gemini-product-lifecycle-plc_batch_known_operators_only] SKIPPED (No server for gemini) [ 33%]
evals/skills/test_eval.py::test_skill[openai-product-lifecycle-plc_proposal_olm_batch_check] PASSED [ 35%]
evals/skills/test_eval.py::test_skill[openai-product-lifecycle-plc_cluster_logging_supported] PASSED [ 38%]
evals/skills/test_eval.py::test_skill[openai-product-lifecycle-plc_web_terminal_compat_check] PASSED [ 40%]
evals/skills/test_eval.py::test_skill[openai-product-lifecycle-plc_compliance_operator_status] PASSED [ 42%]
evals/skills/test_eval.py::test_skill[openai-product-lifecycle-plc_ocp_platform_status] PASSED [ 45%]
evals/skills/test_eval.py::test_skill[openai-product-lifecycle-plc_ocp_old_version_extended] PASSED [ 47%]
evals/skills/test_eval.py::test_skill[openai-product-lifecycle-plc_batch_known_operators_only] PASSED [ 50%]
evals/skills/test_eval.py::test_skill[deepagents-claude-product-lifecycle-plc_proposal_olm_batch_check] SKIPPED (No server for deepagents-claude) [ 52%]
evals/skills/test_eval.py::test_skill[deepagents-claude-product-lifecycle-plc_cluster_logging_supported] SKIPPED (No server for deepagents-claude) [ 54%]
evals/skills/test_eval.py::test_skill[deepagents-claude-product-lifecycle-plc_web_terminal_compat_check] SKIPPED (No server for deepagents-claude) [ 57%]
evals/skills/test_eval.py::test_skill[deepagents-claude-product-lifecycle-plc_compliance_operator_status] SKIPPED (No server for deepagents-claude) [ 59%]
evals/skills/test_eval.py::test_skill[deepagents-claude-product-lifecycle-plc_ocp_platform_status] SKIPPED (No server for deepagents-claude) [ 61%]
evals/skills/test_eval.py::test_skill[deepagents-claude-product-lifecycle-plc_ocp_old_version_extended] SKIPPED (No server for deepagents-claude) [ 64%]
evals/skills/test_eval.py::test_skill[deepagents-claude-product-lifecycle-plc_batch_known_operators_only] SKIPPED (No server for deepagents-claude) [ 66%]
evals/skills/test_eval.py::test_skill[deepagents-gemini-product-lifecycle-plc_proposal_olm_batch_check] SKIPPED (No server for deepagents-gemini) [ 69%]
evals/skills/test_eval.py::test_skill[deepagents-gemini-product-lifecycle-plc_cluster_logging_supported] SKIPPED (No server for deepagents-gemini) [ 71%]
evals/skills/test_eval.py::test_skill[deepagents-gemini-product-lifecycle-plc_web_terminal_compat_check] SKIPPED (No server for deepagents-gemini) [ 73%]
evals/skills/test_eval.py::test_skill[deepagents-gemini-product-lifecycle-plc_compliance_operator_status] SKIPPED (No server for deepagents-gemini) [ 76%]
evals/skills/test_eval.py::test_skill[deepagents-gemini-product-lifecycle-plc_ocp_platform_status] SKIPPED (No server for deepagents-gemini) [ 78%]
evals/skills/test_eval.py::test_skill[deepagents-gemini-product-lifecycle-plc_ocp_old_version_extended] SKIPPED (No server for deepagents-gemini) [ 80%]
evals/skills/test_eval.py::test_skill[deepagents-gemini-product-lifecycle-plc_batch_known_operators_only] SKIPPED (No server for deepagents-gemini) [ 83%]
evals/skills/test_eval.py::test_skill[deepagents-openai-product-lifecycle-plc_proposal_olm_batch_check] SKIPPED (No server for deepagents-openai) [ 85%]
evals/skills/test_eval.py::test_skill[deepagents-openai-product-lifecycle-plc_cluster_logging_supported] SKIPPED (No server for deepagents-openai) [ 88%]
evals/skills/test_eval.py::test_skill[deepagents-openai-product-lifecycle-plc_web_terminal_compat_check] SKIPPED (No server for deepagents-openai) [ 90%]
evals/skills/test_eval.py::test_skill[deepagents-openai-product-lifecycle-plc_compliance_operator_status] SKIPPED (No server for deepagents-openai) [ 92%]
evals/skills/test_eval.py::test_skill[deepagents-openai-product-lifecycle-plc_ocp_platform_status] SKIPPED (No server for deepagents-openai) [ 95%]
evals/skills/test_eval.py::test_skill[deepagents-openai-product-lifecycle-plc_ocp_old_version_extended] SKIPPED (No server for deepagents-openai) [ 97%]
evals/skills/test_eval.py::test_skill[deepagents-openai-product-lifecycle-plc_batch_known_operators_only] SKIPPED (No server for deepagents-openai) [100%]
============================================================================= 7 passed, 35 skipped, 186 deselected in 63.92s (0:01:03) =============================================================================
eval-openai🔍 Expand: Container Logs |
wking
reviewed
Jul 24, 2026
jhadvig
pushed a commit
to jhadvig/agentic-skills
that referenced
this pull request
Jul 24, 2026
Align plc_proposal_olm_batch_check with the new olm-check output from openshift#43: test the version-aware found/not-found split instead of the old all-versions dump. Changes: - Switch cluster-logging to v6.3.1 (tracked but not OCP 4.21 compatible) - Replace operators_with/without_lifecycle_data with three finer counts: api_tracked (4), version_tracked (3), ocp_compatible (2) - Update query to ask for per-version and compatibility breakdown
jrangelramos
force-pushed
the
fix/olm-check-per-operator-verdicts
branch
from
July 24, 2026 11:59
922f52f to
9febc34
Compare
harche
suggested changes
Jul 24, 2026
harche
left a comment
harche
left a comment
Contributor
There was a problem hiding this comment.
Thanks, the per operator verdict approach is a real improvement. I validated this branch against the live API before reviewing: the api_all() switch has value beyond the stated fix, since the old code misses openshift-pipelines-operator-rh entirely while this branch finds it, and the unfiltered endpoint does return all 236 products in one unpaginated response, so that assumption holds. Inline comments follow, one of them is a live crash.
One thing I could not attach inline because the file is not part of this diff: this branch produces the 3/2 split you describe (compliance 1.9, cluster-logging 6.5 and pipelines 1.22 are all tracked today), but the committed eval expects operators_without_lifecycle_data: 3, so it fails even with this fix:
Ground truth drifted since the eval was written (compliance-operator 1.9 got added to the API), so the follow up eval update needs a number change, not just a rerun, and live data drift may keep biting these evals.
Happy to approve once the crash and the lifecycle_unavailable mismatch are addressed, both are small fixes plus test updates.
1 task
jrangelramos
added a commit
to jrangelramos/agentic-skills
that referenced
this pull request
Jul 24, 2026
- Strip leading 'v' prefix in version normalization (v1.16.0 → 1.16) - Match API versions with '.x' suffix (1.8.x matches normalized 1.8) - Guard against missing 'name' key in API version entries to avoid KeyError crash on live data (e.g. skupper-operator) - Derive lifecycle_unavailable from results instead of maintaining a separate list, so the summary always agrees with per-result verdicts - Document both success shapes in SKILL.md (with-version vs no-version) to avoid overpromising fields the no-version path doesn't return Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
jrangelramos
force-pushed
the
fix/olm-check-per-operator-verdicts
branch
from
July 24, 2026 15:15
c4ec733 to
0b4d49d
Compare
Contributor
|
/lgtm |
openshift-ci
Bot
added
lgtm
Contributor
|
/override ci/prow/evals this is cluster update related skill change, so I will leave final approval to @wking /hold |
openshift-ci
Bot
added
the
do-not-merge/hold
|
@harche: /override requires failed status contexts, check run or a prowjob name to operate on.
Only the following failed contexts/checkruns were expected:
If you are trying to override a checkrun that has a space in it, you must put a double quote on the context. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Contributor
|
/override ci/prow/eval |
|
@harche: Overrode contexts on behalf of harche: ci/prow/eval DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
wking
reviewed
Jul 24, 2026
Member
Author
|
The SKILL has been verified and exercised in a 5.0.0ec4 cluster with 2 operators installed: Serverless 1.37 (tracked by PLCC API) and Pipelines 1.23 (not yet tracked on PLCC api). Pod Logs Cluster Update Assessment |
jrangelramos
force-pushed
the
fix/olm-check-per-operator-verdicts
branch
from
July 24, 2026 20:48
0b4d49d to
d36a861
Compare
Member
Author
|
Test logs using evals from #34 w/ claude-opus-4-6 model 🔍 Expand: Eval Run Logs$ CLAUDE_CODE_USE_VERTEX=1 \
ANTHROPIC_VERTEX_PROJECT_ID=itpc-gcp-hcm-pe-eng-claude \
ANTHROPIC_MODEL=claude-opus-4-6 \
CLOUD_ML_REGION=us-east5 \
EVAL_HEALTH_TIMEOUT=180 \
bash evals/run.sh -k "product-lifecycle" --timeout=600
Starting provider containers...
claude: port 18080 (container cf8d6164fa56)
Waiting for servers...
claude: ready
Running evals...
/usr/lib/python3.14/site-packages/pytest_asyncio/plugin.py:211: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset.
The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session"
warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET))
=============================================================================================== test session starts ================================================================================================
platform linux -- Python 3.14.3, pytest-8.3.5, pluggy-1.6.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/jeramos/pixaa/test/test-ocp5/agentic-skills/evals
configfile: pytest.ini
plugins: anyio-4.12.1, asyncio-1.1.0, xdist-3.7.0, timeout-2.4.0
asyncio: mode=Mode.AUTO, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
timeout: 600.0s
timeout method: signal
timeout func_only: False
collected 228 items / 186 deselected / 42 selected
evals/skills/test_eval.py::test_skill[claude-product-lifecycle-plc_proposal_olm_batch_check] PASSED [ 2%]
evals/skills/test_eval.py::test_skill[claude-product-lifecycle-plc_cluster_logging_supported] PASSED [ 4%]
evals/skills/test_eval.py::test_skill[claude-product-lifecycle-plc_web_terminal_compat_check] PASSED [ 7%]
evals/skills/test_eval.py::test_skill[claude-product-lifecycle-plc_compliance_operator_status] PASSED [ 9%]
evals/skills/test_eval.py::test_skill[claude-product-lifecycle-plc_ocp_platform_status] PASSED [ 11%]
evals/skills/test_eval.py::test_skill[claude-product-lifecycle-plc_ocp_old_version_extended] PASSED [ 14%]
evals/skills/test_eval.py::test_skill[claude-product-lifecycle-plc_batch_known_operators_only] PASSED [ 16%]
evals/skills/test_eval.py::test_skill[gemini-product-lifecycle-plc_proposal_olm_batch_check] SKIPPED (No server for gemini) [ 19%]
evals/skills/test_eval.py::test_skill[gemini-product-lifecycle-plc_cluster_logging_supported] SKIPPED (No server for gemini) [ 21%]
evals/skills/test_eval.py::test_skill[gemini-product-lifecycle-plc_web_terminal_compat_check] SKIPPED (No server for gemini) [ 23%]
evals/skills/test_eval.py::test_skill[gemini-product-lifecycle-plc_compliance_operator_status] SKIPPED (No server for gemini) [ 26%]
evals/skills/test_eval.py::test_skill[gemini-product-lifecycle-plc_ocp_platform_status] SKIPPED (No server for gemini) [ 28%]
evals/skills/test_eval.py::test_skill[gemini-product-lifecycle-plc_ocp_old_version_extended] SKIPPED (No server for gemini) [ 30%]
evals/skills/test_eval.py::test_skill[gemini-product-lifecycle-plc_batch_known_operators_only] SKIPPED (No server for gemini) [ 33%]
evals/skills/test_eval.py::test_skill[openai-product-lifecycle-plc_proposal_olm_batch_check] SKIPPED (No server for openai) [ 35%]
evals/skills/test_eval.py::test_skill[openai-product-lifecycle-plc_cluster_logging_supported] SKIPPED (No server for openai) [ 38%]
evals/skills/test_eval.py::test_skill[openai-product-lifecycle-plc_web_terminal_compat_check] SKIPPED (No server for openai) [ 40%]
evals/skills/test_eval.py::test_skill[openai-product-lifecycle-plc_compliance_operator_status] SKIPPED (No server for openai) [ 42%]
evals/skills/test_eval.py::test_skill[openai-product-lifecycle-plc_ocp_platform_status] SKIPPED (No server for openai) [ 45%]
evals/skills/test_eval.py::test_skill[openai-product-lifecycle-plc_ocp_old_version_extended] SKIPPED (No server for openai) [ 47%]
evals/skills/test_eval.py::test_skill[openai-product-lifecycle-plc_batch_known_operators_only] SKIPPED (No server for openai) [ 50%]
evals/skills/test_eval.py::test_skill[deepagents-claude-product-lifecycle-plc_proposal_olm_batch_check] SKIPPED (No server for deepagents-claude) [ 52%]
evals/skills/test_eval.py::test_skill[deepagents-claude-product-lifecycle-plc_cluster_logging_supported] SKIPPED (No server for deepagents-claude) [ 54%]
evals/skills/test_eval.py::test_skill[deepagents-claude-product-lifecycle-plc_web_terminal_compat_check] SKIPPED (No server for deepagents-claude) [ 57%]
evals/skills/test_eval.py::test_skill[deepagents-claude-product-lifecycle-plc_compliance_operator_status] SKIPPED (No server for deepagents-claude) [ 59%]
evals/skills/test_eval.py::test_skill[deepagents-claude-product-lifecycle-plc_ocp_platform_status] SKIPPED (No server for deepagents-claude) [ 61%]
evals/skills/test_eval.py::test_skill[deepagents-claude-product-lifecycle-plc_ocp_old_version_extended] SKIPPED (No server for deepagents-claude) [ 64%]
evals/skills/test_eval.py::test_skill[deepagents-claude-product-lifecycle-plc_batch_known_operators_only] SKIPPED (No server for deepagents-claude) [ 66%]
evals/skills/test_eval.py::test_skill[deepagents-gemini-product-lifecycle-plc_proposal_olm_batch_check] SKIPPED (No server for deepagents-gemini) [ 69%]
evals/skills/test_eval.py::test_skill[deepagents-gemini-product-lifecycle-plc_cluster_logging_supported] SKIPPED (No server for deepagents-gemini) [ 71%]
evals/skills/test_eval.py::test_skill[deepagents-gemini-product-lifecycle-plc_web_terminal_compat_check] SKIPPED (No server for deepagents-gemini) [ 73%]
evals/skills/test_eval.py::test_skill[deepagents-gemini-product-lifecycle-plc_compliance_operator_status] SKIPPED (No server for deepagents-gemini) [ 76%]
evals/skills/test_eval.py::test_skill[deepagents-gemini-product-lifecycle-plc_ocp_platform_status] SKIPPED (No server for deepagents-gemini) [ 78%]
evals/skills/test_eval.py::test_skill[deepagents-gemini-product-lifecycle-plc_ocp_old_version_extended] SKIPPED (No server for deepagents-gemini) [ 80%]
evals/skills/test_eval.py::test_skill[deepagents-gemini-product-lifecycle-plc_batch_known_operators_only] SKIPPED (No server for deepagents-gemini) [ 83%]
evals/skills/test_eval.py::test_skill[deepagents-openai-product-lifecycle-plc_proposal_olm_batch_check] SKIPPED (No server for deepagents-openai) [ 85%]
evals/skills/test_eval.py::test_skill[deepagents-openai-product-lifecycle-plc_cluster_logging_supported] SKIPPED (No server for deepagents-openai) [ 88%]
evals/skills/test_eval.py::test_skill[deepagents-openai-product-lifecycle-plc_web_terminal_compat_check] SKIPPED (No server for deepagents-openai) [ 90%]
evals/skills/test_eval.py::test_skill[deepagents-openai-product-lifecycle-plc_compliance_operator_status] SKIPPED (No server for deepagents-openai) [ 92%]
evals/skills/test_eval.py::test_skill[deepagents-openai-product-lifecycle-plc_ocp_platform_status] SKIPPED (No server for deepagents-openai) [ 95%]
evals/skills/test_eval.py::test_skill[deepagents-openai-product-lifecycle-plc_ocp_old_version_extended] SKIPPED (No server for deepagents-openai) [ 97%]
evals/skills/test_eval.py::test_skill[deepagents-openai-product-lifecycle-plc_batch_known_operators_only] SKIPPED (No server for deepagents-openai) [100%]
============================================================================ 7 passed, 35 skipped, 186 deselected in 740.62s (0:12:20) =============================================================================
eval-claude🔍 Expand: Sandbox LogsINFO lightspeed_agentic.config: Resolved LIGHTSPEED_PROVIDER=anthropic → SDK=claude
INFO lightspeed_agentic.app: Starting app (sdk=claude, model=claude-opus-4-6, LIGHTSPEED_MODEL=, audit=False)
INFO: Started server process [2]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
INFO: 192.168.0.30:35852 - "GET /health HTTP/1.1" 200 OK
INFO lightspeed_agentic: [agent] Starting query (model=claude-opus-4-6, provider=claude, trace_id=60c314a01ebdd3dd2ccef62df212dce9)
INFO claude_agent_sdk._internal.transport.subprocess_cli: Using bundled Claude Code CLI: /opt/app-root/lib64/python3.12/site-packages/claude_agent_sdk/_bundled/claude
INFO lightspeed_agentic: [provider:run] thinking: The user wants me to use the product-lifecycle skill to look up lifecycle data for 5 operators against OCP 4.21. Let me invoke the skill with the appropriate information.
INFO lightspeed_agentic: [provider:run] tool_use: Skill({"skill": "product-lifecycle", "args": "Look up lifecycle data for each of these 5 operators against OCP 4.21:\n\n1. compliance-operator v1.9.0 (channel: stable, package: compliance-operator)\n2. cluster-logging v6.3.1 (channel: stable-6.3, package: cluster-logging)\n3. devworkspace-operator v0.41.0)
INFO lightspeed_agentic: [provider:run] thinking: The skill has loaded and given me instructions on how to use the `plc_lookup.py` script. I need to run the `olm-check` command with all 5 operators and their versions against OCP 4.21.
INFO lightspeed_agentic: [provider:run] tool_use: Bash({"command": "python3 /app/workspace/skills/.claude/skills/product-lifecycle/scripts/plc_lookup.py olm-check --ocp 4.21 --operators '[{\"package\":\"compliance-operator\",\"version\":\"1.9.0\"},{\"package\":\"cluster-logging\",\"version\":\"6.3.1\"},{\"package\":\"devworkspace-operator\",\"version\":)
INFO lightspeed_agentic: [provider:run] thinking: Now let me analyze the results:
1. **compliance-operator v1.9.0** — Found in API, version 1.9 tracked, Full Support, OCP 4.21 compatible ✅
2. **cluster-logging v6.3.1** — Found in API, version 6.3 tracked, Maintenance Support, OCP 4.21 **not compatible** ❌
3. **devworkspace-operator v0.41.0** — NOT found in API (package not found)
4. **openshift-pipelines-operator-rh v1.22.0** — Found in API, version 1.22 tracked, Full Support, OCP 4.21 compatible ✅
5. **web-terminal v1.16.0** — Found in API but version 1.16 isn't tracked (latest available is 1.15)
So out of five operators checked, three packages exist in the API but only two are fully compatible with OCP 4.21, one has a version mismatch, one isn't available at all, and one isn't compatible with the target version.
For the count of operators whose package was found in the PLC API, I need to look at which ones have no error field — that's compliance-operator, cluster-logging, and openshift-pipelines-operator-rh. Web-terminal's package does exist but the version tracking issue complicates whether it counts as "found." dev-workspace-operator also has an error ("package not found in PLC API"), so that confirms api_tracked_count = 3. I'm reconsidering what "no error reported" actually means in the context of the metric—it seems like it should mean both the package was found AND the version matched, since an error is reported whenever there's a version mismatch or missing package. So the error field is the key distinguisher here. Web-terminal has the package in the API but returned an error because that specific version isn't tracked, while devworkspace-operator's package wasn't found at all. The question is whether "no error reported" means I should count web-terminal or not—the phrasing suggests counting based on whether an error occurred, which would give me 3, but the intent might be to count any package that exists in the API regardless of version match, which would be 4. I'm realizing there's ambiguity in how to
INFO lightspeed_agentic: [provider:run] tool_use: StructuredOutput({"olm_check_ran": true, "operators_checked": 5, "operators_api_tracked_count": 4, "operators_version_tracked_count": 3, "operators_ocp_compatible_count": 2})
INFO lightspeed_agentic: [provider:run] result: cost=$0.3029, tokens=0
INFO lightspeed_agentic: [provider:run] output: {"olm_check_ran": true, "operators_checked": 5, "operators_api_tracked_count": 4, "operators_version_tracked_count": 3, "operators_ocp_compatible_count": 2}
INFO lightspeed_agentic: [agent] query complete: success=True, cost=$0.3029
INFO: 192.168.0.30:35862 - "POST /v1/agent/run HTTP/1.1" 200 OK
INFO lightspeed_agentic: [agent] Starting query (model=claude-opus-4-6, provider=claude, trace_id=32e91c2574cd098d6d65e728b210fae9)
INFO claude_agent_sdk._internal.transport.subprocess_cli: Using bundled Claude Code CLI: /opt/app-root/lib64/python3.12/site-packages/claude_agent_sdk/_bundled/claude
INFO lightspeed_agentic: [provider:run] thinking: The user wants me to look up the lifecycle status of cluster-logging using the product-lifecycle skill. Let me invoke that skill.
INFO lightspeed_agentic: [provider:run] tool_use: Skill({"skill": "product-lifecycle", "args": "Look up the product \"cluster-logging\" (package name: cluster-logging) in the Red Hat Product Life Cycle API. Report whether the product was found, its current lifecycle/support status, and whether it is compatible with OCP 4.21."})
INFO lightspeed_agentic: [provider:run] thinking: The skill has loaded and given me instructions on how to use the product-lifecycle tool. I need to run the `olm-check` command with the cluster-logging package and version 6.5.1, checking against OCP 4.21.
INFO lightspeed_agentic: [provider:run] tool_use: Bash({"command": "python3 /app/workspace/skills/.claude/skills/product-lifecycle/scripts/plc_lookup.py olm-check --ocp 4.21 --operators '[{\"package\":\"cluster-logging\",\"version\":\"6.5.1\"}]'", "description": "Look up cluster-logging v6.5.1 lifecycle status and OCP 4.21 compatibility"})
INFO lightspeed_agentic: [provider:run] tool_use: StructuredOutput({"product_found": true, "status": "supported", "ocp_compatible": true})
INFO lightspeed_agentic: [provider:run] result: cost=$0.2222, tokens=0
INFO lightspeed_agentic: [provider:run] output: {"product_found": true, "status": "supported", "ocp_compatible": true}
INFO lightspeed_agentic: [agent] query complete: success=True, cost=$0.2222
INFO: 192.168.0.30:44702 - "POST /v1/agent/run HTTP/1.1" 200 OK
INFO lightspeed_agentic: [agent] Starting query (model=claude-opus-4-6, provider=claude, trace_id=650fa9f6514ac7542669d850d8e78897)
INFO claude_agent_sdk._internal.transport.subprocess_cli: Using bundled Claude Code CLI: /opt/app-root/lib64/python3.12/site-packages/claude_agent_sdk/_bundled/claude
INFO lightspeed_agentic: [provider:run] thinking: The user wants me to check the product lifecycle status of web-terminal for OCP 4.21 compatibility. Let me use the product-lifecycle skill.
INFO lightspeed_agentic: [provider:run] tool_use: Skill({"skill": "product-lifecycle", "args": "Check if the product \"web-terminal\" (package name: web-terminal) has any version that is both \"supported\" (full support phase) and compatible with OCP 4.21. Report all versions found, their support phases, and OCP compatibility ranges."})
INFO lightspeed_agentic: [provider:run] thinking: Let me first look up all versions of web-terminal to see what's available, then check compatibility with OCP 4.21.
INFO lightspeed_agentic: [provider:run] tool_use: Bash({"command": "python3 /app/workspace/skills/.claude/skills/product-lifecycle/scripts/plc_lookup.py products \"web-terminal\" --ocp 4.21", "description": "Look up web-terminal product lifecycle with OCP 4.21 compatibility"})
INFO lightspeed_agentic: [provider:run] tool_use: Bash({"command": "python3 /app/workspace/skills/.claude/skills/product-lifecycle/scripts/plc_lookup.py olm-check --ocp 4.21 --operators '[{\"package\":\"web-terminal\"}]'", "description": "Check web-terminal operator via OLM package lookup"})
INFO lightspeed_agentic: [provider:run] thinking: The web-terminal package was found, but v1.16 is not tracked. The available versions go up to 1.15. Let me check the installed version (1.16) and also check all available versions for OCP 4.21 compatibility and support status. Let me check v1.15 (the latest tracked version) specifically.
INFO lightspeed_agentic: [provider:run] tool_use: Bash({"command": "python3 /app/workspace/skills/.claude/skills/product-lifecycle/scripts/plc_lookup.py olm-check --ocp 4.21 --operators '[{\"package\":\"web-terminal\",\"version\":\"1.15\"},{\"package\":\"web-terminal\",\"version\":\"1.14\"},{\"package\":\"web-terminal\",\"version\":\"1.13\"},{\"package\)
INFO lightspeed_agentic: [provider:run] thinking: Let me summarize the findings:
1. **web-terminal** was found in the Product Lifecycle API ✅
2. The installed version is **v1.16.0**, but the API only tracks versions up to **v1.15**. Version 1.16 is not tracked.
3. Looking at all tracked versions and their OCP 4.21 compatibility:
| Version | Status | OCP 4.21 Compatible |
|---------|--------|-------------------|
| 1.15 | Full Support | ❌ false |
| 1.14 | Maintenance Support | ❌ false |
| 1.13 | Maintenance Support | ❌ false |
| 1.12 | End of life | ❌ false |
| 1.11 | Extended Support | ❌ false |
| 1.10 | End of life | ❌ false |
None of the tracked versions are compatible with OCP 4.21, and the only one in full support (v1.15) doesn't meet that requirement either. So the product exists, but there's no compatible version available.
INFO lightspeed_agentic: [provider:run] tool_use: StructuredOutput({"product_found": true, "has_supported_version_for_421": false})
INFO lightspeed_agentic: [provider:run] result: cost=$0.2886, tokens=0
INFO lightspeed_agentic: [provider:run] output: {"product_found": true, "has_supported_version_for_421": false}
INFO lightspeed_agentic: [agent] query complete: success=True, cost=$0.2886
INFO: 192.168.0.30:56794 - "POST /v1/agent/run HTTP/1.1" 200 OK
INFO lightspeed_agentic: [agent] Starting query (model=claude-opus-4-6, provider=claude, trace_id=19201076bd39d5e819689e62d52da876)
INFO claude_agent_sdk._internal.transport.subprocess_cli: Using bundled Claude Code CLI: /opt/app-root/lib64/python3.12/site-packages/claude_agent_sdk/_bundled/claude
INFO lightspeed_agentic: [provider:run] thinking: The user wants me to check the lifecycle status of compliance-operator v1.9 using the product-lifecycle skill.
INFO lightspeed_agentic: [provider:run] tool_use: Skill({"skill": "product-lifecycle", "args": "Check the lifecycle status of compliance-operator v1.9 (package name: compliance-operator). Report whether the product was found in the Red Hat Product Life Cycle API and its current lifecycle phase/status."})
INFO lightspeed_agentic: [provider:run] tool_use: Bash({"command": "python3 /app/workspace/skills/.claude/skills/product-lifecycle/scripts/plc_lookup.py olm-check --operators '[{\"package\":\"compliance-operator\",\"version\":\"1.9.0\"}]'", "description": "Check lifecycle status of compliance-operator v1.9"})
INFO lightspeed_agentic: [provider:run] tool_use: Bash({"command": "python3 /app/workspace/skills/.claude/skills/product-lifecycle/scripts/plc_lookup.py products \"compliance operator\" 2>&1 | head -100", "description": "Look up compliance-operator product lifecycle"})
INFO lightspeed_agentic: [provider:run] tool_use: StructuredOutput({"product_found": true, "status": "supported"})
INFO lightspeed_agentic: [provider:run] result: cost=$0.2364, tokens=0
INFO lightspeed_agentic: [provider:run] output: {"product_found": true, "status": "supported"}
INFO lightspeed_agentic: [agent] query complete: success=True, cost=$0.2364
INFO: 192.168.0.30:51664 - "POST /v1/agent/run HTTP/1.1" 200 OK
INFO lightspeed_agentic: [agent] Starting query (model=claude-opus-4-6, provider=claude, trace_id=c7aa718ebe85a0508c5d0b4efcac66b2)
INFO claude_agent_sdk._internal.transport.subprocess_cli: Using bundled Claude Code CLI: /opt/app-root/lib64/python3.12/site-packages/claude_agent_sdk/_bundled/claude
INFO lightspeed_agentic: [provider:run] thinking: The user wants me to check the lifecycle status of OCP 4.21 using the product-lifecycle skill. Let me invoke that skill.
INFO lightspeed_agentic: [provider:run] tool_use: Skill({"skill": "product-lifecycle", "args": "Look up \"Red Hat OpenShift Container Platform\" version 4.21 in the Red Hat Product Life Cycle API. Report whether the product was found and what the current lifecycle/support phase is for version 4.21."})
INFO lightspeed_agentic: [provider:run] thinking: The skill has loaded and given me instructions on how to use the product-lifecycle tool. I need to run the Python script to look up "Red Hat OpenShift Container Platform" and check version 4.21's lifecycle status.
INFO lightspeed_agentic: [provider:run] tool_use: Bash({"command": "python3 /app/workspace/skills/.claude/skills/product-lifecycle/scripts/plc_lookup.py products \"Red Hat OpenShift Container Platform\"", "description": "Look up OCP lifecycle status via Red Hat Product Life Cycle API"})
INFO lightspeed_agentic: [provider:run] tool_use: Bash({"command": "python3 -c \"\nimport json\nwith open('/home/agent/.claude/projects/-app-workspace-skills/519baa64-9a46-49b7-a7a0-5df8285f625f/tool-results/bpu65mu73.txt') as f:\n data = json.load(f)\nfor r in data['results']:\n if r['version'] == '4.21':\n print(json.dumps(r, indent=2))\n)
INFO lightspeed_agentic: [provider:run] tool_use: StructuredOutput({"product_found": true, "status": "supported"})
INFO lightspeed_agentic: [provider:run] result: cost=$0.2507, tokens=0
INFO lightspeed_agentic: [provider:run] output: {"product_found": true, "status": "supported"}
INFO lightspeed_agentic: [agent] query complete: success=True, cost=$0.2507
INFO: 192.168.0.30:56520 - "POST /v1/agent/run HTTP/1.1" 200 OK
INFO lightspeed_agentic: [agent] Starting query (model=claude-opus-4-6, provider=claude, trace_id=5135fc9185730e951e38aaec86a44bfc)
INFO claude_agent_sdk._internal.transport.subprocess_cli: Using bundled Claude Code CLI: /opt/app-root/lib64/python3.12/site-packages/claude_agent_sdk/_bundled/claude
INFO lightspeed_agentic: [provider:run] thinking: The user wants me to check the lifecycle status of Red Hat OpenShift Container Platform version 4.14 using the product-lifecycle skill.
INFO lightspeed_agentic: [provider:run] tool_use: Skill({"skill": "product-lifecycle", "args": "Check the lifecycle status of Red Hat OpenShift Container Platform version 4.14. Report its current support phase and any relevant dates (GA, end of full support, end of maintenance, end of life)."})
INFO lightspeed_agentic: [provider:run] thinking: Let me run the product lifecycle lookup tool to check OCP 4.14's status.
INFO lightspeed_agentic: [provider:run] tool_use: Bash({"command": "python3 /app/workspace/skills/.claude/skills/product-lifecycle/scripts/plc_lookup.py products \"Red Hat OpenShift Container Platform\"", "description": "Query Red Hat Product Life Cycle API for OCP"})
INFO lightspeed_agentic: [provider:run] tool_use: Bash({"command": "python3 /app/workspace/skills/.claude/skills/product-lifecycle/scripts/plc_lookup.py products \"Red Hat OpenShift Container Platform\" | python3 -c \"\nimport json, sys\ndata = json.load(sys.stdin)\nfor r in data['results']:\n if r['version'] == '4.14':\n print(json.dumps(r, i)
INFO lightspeed_agentic: [provider:run] thinking: I found OCP 4.14. Let me analyze the data:
- Product: Red Hat OpenShift Container Platform
- Version: 4.14
- Current status: "Extended Support"
- Today's date: 2026-07-24
Phases:
1. GA: ended 2023-10-31
2. Full Support: 2023-10-31 → 2024-05-27
3. Maintenance Support: 2024-05-28 → 2025-05-01
4. Extended Update Support: 2025-05-02 → 2025-10-31
5. Extended Update Support Term 2: 2025-11-01 → 2026-10-31
6. Extended Update Support Term 3: 2026-11-01 → 2027-10-31
So as of today, OCP 4.14 is currently in Extended Update Support Term 2, which aligns with the API's "Extended Support" status. I need to normalize the status field to match the actual phase we're in.
INFO lightspeed_agentic: [provider:run] tool_use: StructuredOutput({"product_found": true, "status": "extended"})
INFO lightspeed_agentic: [provider:run] result: cost=$0.2567, tokens=0
INFO lightspeed_agentic: [provider:run] output: {"product_found": true, "status": "extended"}
INFO lightspeed_agentic: [agent] query complete: success=True, cost=$0.2567
INFO: 192.168.0.30:41606 - "POST /v1/agent/run HTTP/1.1" 200 OK
INFO lightspeed_agentic: [agent] Starting query (model=claude-opus-4-6, provider=claude, trace_id=71f5475a0d423c18bf924eebf115afd4)
INFO claude_agent_sdk._internal.transport.subprocess_cli: Using bundled Claude Code CLI: /opt/app-root/lib64/python3.12/site-packages/claude_agent_sdk/_bundled/claude
INFO lightspeed_agentic: [provider:run] thinking: The user wants me to check the lifecycle status of two operators against OCP 4.21 using the product-lifecycle skill. Let me invoke that skill.
INFO lightspeed_agentic: [provider:run] tool_use: Skill({"skill": "product-lifecycle", "args": "Check lifecycle status for these 2 operators against OCP 4.21: 1. cluster-logging (package: cluster-logging) 2. web-terminal (package: web-terminal). For each operator, look up its lifecycle data and report whether both were found in the API and whether any of)
INFO lightspeed_agentic: [provider:run] thinking: Let me run the olm-check command to look up both operators against OCP 4.21.
INFO lightspeed_agentic: [provider:run] tool_use: Bash({"command": "python3 /app/workspace/skills/.claude/skills/product-lifecycle/scripts/plc_lookup.py olm-check --ocp 4.21 --operators '[{\"package\":\"cluster-logging\"},{\"package\":\"web-terminal\"}]'", "description": "Look up lifecycle data for cluster-logging and web-terminal operators against OCP )
INFO lightspeed_agentic: [provider:run] thinking: Both operators were found (no errors, lifecycle_unavailable is empty), but since no specific version was provided, the API returned `available_versions` instead of lifecycle status. I need to check which versions are compatible with OCP 4.21 and whether any are end-of-life. Let me query each operator's versions to find which ones are compatible with OCP 4.21, and then check their EOL status.
Let me look up each product with the --ocp flag to see which versions are compatible with OCP 4.21.
INFO lightspeed_agentic: [provider:run] tool_use: Bash({"command": "python3 /app/workspace/skills/.claude/skills/product-lifecycle/scripts/plc_lookup.py products \"logging for Red Hat OpenShift\" --ocp 4.21", "description": "Look up cluster-logging lifecycle data with OCP 4.21 compatibility"})
INFO lightspeed_agentic: [provider:run] tool_use: Bash({"command": "python3 /app/workspace/skills/.claude/skills/product-lifecycle/scripts/plc_lookup.py products \"web terminal operator\" --ocp 4.21", "description": "Look up web-terminal lifecycle data with OCP 4.21 compatibility"})
INFO lightspeed_agentic: [provider:run] thinking: Now let me analyze the results:
**cluster-logging** (Logging for Red Hat OpenShift):
- Both found in API: Yes
- Versions compatible with OCP 4.21 (`ocp_compatible: true`):
- 6.6 — Full Support
- 6.5 — Full Support
- 6.4 — Full Support
- None of the OCP 4.21-compatible versions are end-of-life. They are all in Full Support.
**web-terminal** (Web Terminal Operator):
- Found in API: Yes
- Versions compatible with OCP 4.21 (`ocp_compatible: true`): **NONE**
- All versions have `ocp_compatible: false`, with the latest (1.15) only supporting OCP 4.20
So for cluster-logging, all three OCP 4.21-compatible versions are in Full Support with no end-of-life versions. For web-terminal, there are no compatible versions at all for OCP 4.21.
Since the question asks whether any OCP 4.21-compatible versions are end-of-life, the answer is false — cluster-logging has none that are EOL, and web-terminal has no compatible versions to begin with.
INFO lightspeed_agentic: [provider:run] tool_use: StructuredOutput({"check_completed": true, "both_found": true, "any_eol": false})
INFO lightspeed_agentic: [provider:run] result: cost=$0.3459, tokens=0
INFO lightspeed_agentic: [provider:run] output: {"check_completed": true, "both_found": true, "any_eol": false}
INFO lightspeed_agentic: [agent] query complete: success=True, cost=$0.3459
INFO: 192.168.0.30:57040 - "POST /v1/agent/run HTTP/1.1" 200 OK
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [2]
|
Member
Author
|
/verified by Jefferson Ramos |
Contributor
|
/override ci/prow/eval |
|
@harche: Overrode contexts on behalf of harche: ci/prow/eval DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
4 tasks
|
/lgtm |
wking
reviewed
Jul 28, 2026
wking
reviewed
Jul 28, 2026
wking
reviewed
Jul 28, 2026
The olm-check command previously dumped all version entries for every
matched package, leaving the LLM to determine whether the specific
installed version was actually tracked. Claude handled this correctly
but gemini and openai miscounted — they saw data for a package and
assumed lifecycle data was available, even when the installed version
was not tracked (e.g. web-terminal v1.16 is not in the API).
Refactor olm-check to return one result per operator. When a version
is provided, the tool normalizes it to major.minor and matches against
API version names. Use a single `error` string field — present means
failure (self-describing), absent means success — so incoherent states
are impossible.
Also switch from api_search("OpenShift") to api_all() so operators
with non-OpenShift product names (e.g. compliance-operator) are not
missed by the name filter.
jrangelramos
force-pushed
the
fix/olm-check-per-operator-verdicts
branch
from
July 28, 2026 19:41
d36a861 to
6b825f3
Compare
openshift-ci-robot
removed
the
verified
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: harche, jrangelramos, wking The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@wking: wking unauthorized: /override is restricted to Repo administrators, approvers in top level OWNERS file, and the following github teams:openshift: openshift-release-oversight openshift-staff-engineers openshift-sustaining-engineers. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Member
|
Also lifting the hold that was waiting on me: /hold cancel |
openshift-ci
Bot
removed
the
do-not-merge/hold
Member
Author
|
/verified by Jefferson Ramos |
openshift-ci-robot
added
the
verified
|
@jrangelramos: This PR has been marked as verified by DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
Contributor
Contributor
|
/override ci/prow/eval |
|
@harche: Overrode contexts on behalf of harche: ci/prow/eval DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@jrangelramos: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
jrangelramos
added a commit
to jhadvig/agentic-skills
that referenced
this pull request
Jul 30, 2026
Align plc_proposal_olm_batch_check with the new olm-check output from openshift#43: test the version-aware found/not-found split instead of the old all-versions dump. Changes: - Switch cluster-logging to v6.3.1 (tracked but not OCP 4.21 compatible) - Replace operators_with/without_lifecycle_data with three finer counts: api_tracked (4), version_tracked (3), ocp_compatible (2) - Update query to ask for per-version and compatibility breakdown
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
olm-checkto return one result per operator with a singleerrorstring field (present = failure, absent = success) instead ofdumping all version entries for matched packages
versionfield to operator input for per-version matchingv1.9.0→1.9): strip leadingv,truncate to major.minor, match
.xsuffixes in API version namesnamekey in API version entries (live API crash)lifecycle_unavailablefrom per-result verdicts so the summaryalways agrees with the details
api_search("OpenShift")toapi_all()to cover alloperators regardless of product name
no-version) and the
errorfield conventionProblem
Based on evals test run from #34 it was possible to verify inconsistencies
between how different models deal with lifecycle data count.
The gemini and openai providers failed
plc_proposal_olm_batch_checkreturning 4/1 instead of the expected 3/2 split. Root cause: the tool
returned all version entries per package, and the LLMs had to infer
whether the installed version was tracked. web-terminal v1.16 is not
in the API (only up to v1.15), but the models saw web-terminal data
and counted it as "has lifecycle data."
Proposal Fix
The tool now answers the question directly. For each operator:
error, withstatus,ocp_compatible,phaseserrorset, withavailable_versionserrorseterrorif the package exists, withproductand
available_versions(nostatus/ocp_compatible/phases)Before / After
CLI call
Before — no way to specify installed version:
python3 plc_lookup.py olm-check --ocp 4.21 \ --operators '[{"package":"cluster-logging"},{"package":"web-terminal"}]'After — optional
versionfield per operator:python3 plc_lookup.py olm-check --ocp 4.21 \ --operators '[{"package":"cluster-logging","version":"6.5.1"},{"package":"web-terminal","version":"1.16.0"}]'Sample output — Before
The old output dumped every version entry for each matched package.
The LLM had to figure out which (if any) matched the installed version:
{ "ocp_target": "4.21", "operators_checked": 2, "lifecycle_unavailable": [], "results": [ { "product": "logging for Red Hat OpenShift", "package": "cluster-logging", "version": "6.5", "status": "Full Support", "ocp_versions": ["4.19", "4.20", "4.21"], "ocp_compatible": true, "phases": [...] }, { "product": "logging for Red Hat OpenShift", "package": "cluster-logging", "version": "5.9", "status": "End of life", "ocp_versions": ["4.13", "4.14", "4.15", "4.16"], "ocp_compatible": false, "phases": [...] }, { "product": "Red Hat OpenShift Web Terminal", "package": "web-terminal", "version": "1.15", "status": "Full Support", "ocp_versions": ["4.19", "4.20", "4.21"], "ocp_compatible": true, "phases": [...] } ] }Sample output — After
One result per operator with a clear verdict.
No
errorfield means success — no ambiguity for the model to resolve:{ "ocp_target": "4.21", "operators_checked": 2, "lifecycle_unavailable": ["web-terminal"], "results": [ { "package": "cluster-logging", "requested_version": "6.5", "product": "logging for Red Hat OpenShift", "status": "Full Support", "ocp_compatible": true, "phases": [...] }, { "package": "web-terminal", "requested_version": "1.16", "error": "version 1.16 not tracked", "available_versions": ["1.15"] } ] }Files changed
cluster-update/product-lifecycle/scripts/plc_lookup.py— core refactorcluster-update/product-lifecycle/SKILL.md— updated documentationcluster-update/product-lifecycle/scripts/tests/test_plc_lookup.py— rewritten testsTest plan
python3 -m pytest cluster-update/product-lifecycle/scripts/tests/ -v)