cloud-provider-kubevirt: move release-4.18..4.22 conformance jobs to Azure by qinqon · Pull Request #83031 · openshift/release

qinqon · 2026-08-06T11:54:45Z

Follow-up to #82940, which did the same for main. Moves the conformance jobs on release-4.18 … release-4.22 from the baremetalds workflow to the Azure conformance workflow, and narrows them to the tests relevant to this component.

Current state

Every stable branch is red on every recent run:

branch	recent runs	ODF step	total conformance failures
release-4.18	7/7 FAILURE	SUCCESS	82
release-4.19	9/9 FAILURE	SUCCESS	many
release-4.20	11/11 FAILURE	SUCCESS	3
release-4.21	3/3 FAILURE	SUCCESS	3
release-4.22	2/2 FAILURE	SUCCESS	3

Note the difference from #82940: ODF is not broken here. hypershift-kubevirt-baremetalds-odf succeeds on every stable branch — the ODF-to-OCP skew that breaks main is specific to the 5.0 platform version, and ODF 4.20 works fine on 4.21/4.22.

What fails is the conformance step, on things unrelated to KCCM. release-4.22 is representative:

[sig-kubevirt] migration ... should maintain node readiness
[sig-node] Managed cluster should verify that nodes have no unexpected reboots [Late]

and 4.20/4.21 are [sig-builds] flakes plus the same migration test.

What this changes

Each branch gets the same block as main:

- as: hypershift-kubevirt-conformance-azure
  optional: true
  steps:
    cluster_profile: openshift-org-azure
    env:
      TEST_ARGS: --run \[sig-kubevirt\].services|LoadBalancer|terminating.endpoints
    workflow: hypershift-kubevirt-azure-conformance

The baremetalds-specific pins (ODF_OPERATOR_SUB_CHANNEL, REDHAT_OPERATORS_INDEX_TAG, LVM_*, METALLB_*, LOCAL_STORAGE_*) are dropped — the Azure workflow's hypershift-kubevirt-install-odf resolves the ODF catalog dynamically, which is what stops this rotting on each OCP bump.

Test selection

The regex selects 13 tests, covering all four scenarios that drive the cloud provider's EnsureLoadBalancer:

#	test
1-7	`[sig-kubevirt] services` (2x LoadBalancer, 3x NodePort, 2x direct pod-to-pod)
8-12	`[sig-network] Services ... terminating endpoints` x5
13	`[sig-network] Services ... on Type LoadBalancer` (multiprotocol)

KCCM's node controller is covered structurally: without it, nodes keep the node.cloudprovider.kubernetes.io/uninitialized taint and never go Ready, so hypershift-kubevirt-health-check fails before the tests start.

[sig-kubevirt] migration is deliberately excluded. The migration itself succeeds and kubevirt-cloud-controller-manager is not involved — but every in-guest client loses its API-server connection ~45s after its own migration completes with http2: client connection lost (matching client-go's HTTP/2 health check window), so the kubelet lease lapses and the node briefly flaps NotReady. Same symptom as OCPBUGS-57594 / OCPBUGS-57642 on other CNIs. The hypershift Azure periodics continue to run that test daily, so the signal is not lost.

Expected outcome

I checked the 13 selected tests against the latest baremetalds run of each branch:

branch	result for the 13 selected tests
release-4.19	13/13 pass
release-4.20	13/13 pass
release-4.21	13/13 pass
release-4.22	13/13 pass
release-4.18	8 pass / 5 fail

So 4.19-4.22 are expected to go green.

release-4.18 is a known exception. The five failures there are precisely the [sig-kubevirt] services tests that cross guest nodes, while both infra-cluster ones pass — a guest-to-guest networking problem, not test noise. This PR does not address it and the job is expected to stay red on 4.18 until that is investigated separately. I've kept 4.18 in scope so all branches share one configuration; happy to drop it if reviewers would rather not carry a known-red job.

Notes for reviewers

hypershift-kubevirt-azure-conformance is already exercised across this whole version range by the openshift/hypershift periodics, including a SUCCESS on release-4.18, so the workflow itself is proven on these payloads.
On baremetalds the workflow set TEST_SKIPS, which makes the conformance chain append --file /tmp/tests. The Azure workflow sets no TEST_SKIPS, so TEST_ARGS passes straight through. Either way --run and --file AND together, so the selection is the same 13 tests.
Jobs are optional: true while they establish a track record; they can be made blocking in a follow-up.
release-4.23 is left alone here — it still carries the stale ODF 4.20 / v4.20 pins and has no job history at all, so it needs its own look.
The openshift-priv mirrors will be regenerated by automation.

/cc @openshift/openshift-team-hypershift

Summary by CodeRabbit

Moves KubeVirt conformance jobs for releases 4.18–4.22 from the bare-metal workflow to the Azure workflow.
Uses the openshift-org-azure cluster profile and targets 13 KCCM load-balancer scenarios.
Removes bare-metal-specific ODF, LVM, MetalLB, and local-storage settings.
Keeps the jobs optional while gathering reliability data. Release 4.18 remains expected to fail because of guest-to-guest networking issues.

…Azure The baremetalds conformance jobs on the stable branches of openshift/cloud-provider-kubevirt have failed every recent run: release-4.18 7/7, release-4.19 9/9, release-4.20 11/11, release-4.21 3/3 and release-4.22 2/2. Unlike main, the failures here are not caused by ODF: the hypershift-kubevirt-baremetalds-odf step succeeds on every stable branch, because the ODF-to-OCP version skew that breaks main is specific to the 5.0 platform version. What fails on these branches is the conformance step, and the failures are almost entirely unrelated to this component. On release-4.20, 4.21 and 4.22 there are only three failures per run, dominated by [sig-builds] flakes and by [sig-kubevirt] migration ... should maintain node readiness, which is a platform issue tracked separately: the live migration itself succeeds and the kubevirt-cloud-controller-manager is not involved, but in-guest clients lose their API server connection for roughly 45 seconds afterwards with "http2: client connection lost", so the node briefly flaps NotReady. Move these jobs to hypershift-kubevirt-azure-conformance, matching what was done for main, and narrow them to the tests relevant to this component with TEST_ARGS=--run. The regex selects 13 tests: the seven [sig-kubevirt] services tests plus the generic [sig-network] Type LoadBalancer and terminating-endpoints tests. Together these cover all four scenarios that drive the cloud provider's EnsureLoadBalancer. The node controller is covered structurally, since without it nodes keep the node.cloudprovider.kubernetes.io/uninitialized taint and the hypershift-kubevirt-health-check step fails before the tests run. All 13 selected tests already pass on release-4.19, 4.20, 4.21 and 4.22 in the most recent baremetalds run of each, so these jobs are expected to go green. release-4.18 is a known exception: five of the seven [sig-kubevirt] services tests fail there, specifically the ones that cross guest nodes, while both infra-cluster ones pass. That is a guest-to-guest networking problem which this change does not address and which needs its own investigation, so the job is expected to remain red on that branch until it is fixed. The hypershift-kubevirt-azure-conformance workflow is already exercised on this whole version range by the openshift/hypershift periodics, including a successful run on release-4.18. The jobs are marked optional while they establish a track record on the new platform. Assisted-By: Claude Opus 4.5 <noreply@anthropic.com> Signed-off-by: Enrique Llorente <ellorent@redhat.com>

openshift-ci · 2026-08-06T11:54:58Z

@qinqon: GitHub didn't allow me to request PR reviews from the following users: openshift/openshift-team-hypershift.

Note that only openshift members and repo collaborators can review this PR, and authors cannot review their own PRs.

Details

In response to this:

Follow-up to #82940, which did the same for main. Moves the conformance jobs on release-4.18 … release-4.22 from the baremetalds workflow to the Azure conformance workflow, and narrows them to the tests relevant to this component.

Current state

Every stable branch is red on every recent run:

branch recent runs ODF step total conformance failures

release-4.18 7/7 FAILURE SUCCESS 82

release-4.19 9/9 FAILURE SUCCESS many

release-4.20 11/11 FAILURE SUCCESS 3

release-4.21 3/3 FAILURE SUCCESS 3

release-4.22 2/2 FAILURE SUCCESS 3

Note the difference from #82940: ODF is not broken here. hypershift-kubevirt-baremetalds-odf succeeds on every stable branch — the ODF-to-OCP skew that breaks main is specific to the 5.0 platform version, and ODF 4.20 works fine on 4.21/4.22.

What fails is the conformance step, on things unrelated to KCCM. release-4.22 is representative:
[sig-kubevirt] migration ... should maintain node readiness
[sig-node] Managed cluster should verify that nodes have no unexpected reboots [Late]
and 4.20/4.21 are [sig-builds] flakes plus the same migration test.

What this changes

Each branch gets the same block as main:
- as: hypershift-kubevirt-conformance-azure
 optional: true
 steps:
   cluster_profile: openshift-org-azure
   env:
     TEST_ARGS: --run \[sig-kubevirt\].services|LoadBalancer|terminating.endpoints
   workflow: hypershift-kubevirt-azure-conformance
The baremetalds-specific pins (ODF_OPERATOR_SUB_CHANNEL, REDHAT_OPERATORS_INDEX_TAG, LVM_*, METALLB_*, LOCAL_STORAGE_*) are dropped — the Azure workflow's hypershift-kubevirt-install-odf resolves the ODF catalog dynamically, which is what stops this rotting on each OCP bump.

Test selection

The regex selects 13 tests, covering all four scenarios that drive the cloud provider's EnsureLoadBalancer:

# test

1-7 [sig-kubevirt] services (2x LoadBalancer, 3x NodePort, 2x direct pod-to-pod)

8-12 [sig-network] Services ... terminating endpoints x5

13 [sig-network] Services ... on Type LoadBalancer (multiprotocol)

KCCM's node controller is covered structurally: without it, nodes keep the node.cloudprovider.kubernetes.io/uninitialized taint and never go Ready, so hypershift-kubevirt-health-check fails before the tests start.

[sig-kubevirt] migration is deliberately excluded. The migration itself succeeds and kubevirt-cloud-controller-manager is not involved — but every in-guest client loses its API-server connection ~45s after its own migration completes with http2: client connection lost (matching client-go's HTTP/2 health check window), so the kubelet lease lapses and the node briefly flaps NotReady. Same symptom as OCPBUGS-57594 / OCPBUGS-57642 on other CNIs. The hypershift Azure periodics continue to run that test daily, so the signal is not lost.

Expected outcome

I checked the 13 selected tests against the latest baremetalds run of each branch:

branch result for the 13 selected tests

release-4.19 13/13 pass

release-4.20 13/13 pass

release-4.21 13/13 pass

release-4.22 13/13 pass

release-4.18 8 pass / 5 fail

So 4.19-4.22 are expected to go green.

release-4.18 is a known exception. The five failures there are precisely the [sig-kubevirt] services tests that cross guest nodes, while both infra-cluster ones pass — a guest-to-guest networking problem, not test noise. This PR does not address it and the job is expected to stay red on 4.18 until that is investigated separately. I've kept 4.18 in scope so all branches share one configuration; happy to drop it if reviewers would rather not carry a known-red job.

Notes for reviewers

hypershift-kubevirt-azure-conformance is already exercised across this whole version range by the openshift/hypershift periodics, including a SUCCESS on release-4.18, so the workflow itself is proven on these payloads.

On baremetalds the workflow set TEST_SKIPS, which makes the conformance chain append --file /tmp/tests. The Azure workflow sets no TEST_SKIPS, so TEST_ARGS passes straight through. Either way --run and --file AND together, so the selection is the same 13 tests.

Jobs are optional: true while they establish a track record; they can be made blocking in a follow-up.

release-4.23 is left alone here — it still carries the stale ODF 4.20 / v4.20 pins and has no job history at all, so it needs its own look.

The openshift-priv mirrors will be regenerated by automation.

/cc @openshift/openshift-team-hypershift

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

coderabbitai · 2026-08-06T11:55:13Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 4e6c572b-d48e-4072-82dc-fc6fe2c68921

📥 Commits

Reviewing files that changed from the base of the PR and between 3b80bac and ad92a10.

⛔ Files ignored due to path filters (5)

ci-operator/jobs/openshift/cloud-provider-kubevirt/openshift-cloud-provider-kubevirt-release-4.18-presubmits.yaml is excluded by !ci-operator/jobs/**
ci-operator/jobs/openshift/cloud-provider-kubevirt/openshift-cloud-provider-kubevirt-release-4.19-presubmits.yaml is excluded by !ci-operator/jobs/**
ci-operator/jobs/openshift/cloud-provider-kubevirt/openshift-cloud-provider-kubevirt-release-4.20-presubmits.yaml is excluded by !ci-operator/jobs/**
ci-operator/jobs/openshift/cloud-provider-kubevirt/openshift-cloud-provider-kubevirt-release-4.21-presubmits.yaml is excluded by !ci-operator/jobs/**
ci-operator/jobs/openshift/cloud-provider-kubevirt/openshift-cloud-provider-kubevirt-release-4.22-presubmits.yaml is excluded by !ci-operator/jobs/**

📒 Files selected for processing (5)

ci-operator/config/openshift/cloud-provider-kubevirt/openshift-cloud-provider-kubevirt-release-4.18.yaml
ci-operator/config/openshift/cloud-provider-kubevirt/openshift-cloud-provider-kubevirt-release-4.19.yaml
ci-operator/config/openshift/cloud-provider-kubevirt/openshift-cloud-provider-kubevirt-release-4.20.yaml
ci-operator/config/openshift/cloud-provider-kubevirt/openshift-cloud-provider-kubevirt-release-4.21.yaml
ci-operator/config/openshift/cloud-provider-kubevirt/openshift-cloud-provider-kubevirt-release-4.22.yaml

Walkthrough

The release 4.18–4.22 CI configurations replace HyperShift KubeVirt bare-metal conformance tests with optional Azure-based tests. The configurations use the openshift-org-azure profile, targeted KubeVirt test arguments, and the Azure conformance workflow.

Changes

Azure KubeVirt conformance

Layer / File(s)	Summary
Update release conformance configurations `ci-operator/config/openshift/cloud-provider-kubevirt/openshift-cloud-provider-kubevirt-release-4.18.yaml`, `...-4.19.yaml`, `...-4.20.yaml`, `...-4.21.yaml`, `...-4.22.yaml`	Replaces bare-metal HyperShift KubeVirt conformance settings with optional Azure configurations. The new settings select the Azure cluster profile, targeted KubeVirt services, LoadBalancer, and terminating-endpoints tests, and the Azure conformance workflow.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

openshift/release#82940: Updates HyperShift KubeVirt conformance configuration from bare-metal to Azure workflows with related profile and test selection changes.

🚥 Pre-merge checks | ✅ 15

✅ Passed checks (15 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes moving the release 4.18–4.22 conformance jobs to Azure.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The PR changes only static CI YAML and generated job metadata; no added Ginkgo title declarations or dynamic test-title values were found.
Test Structure And Quality	✅ Passed	The PR changes only CI YAML configuration and generated job YAML; it adds no Ginkgo It blocks or test code to review against these requirements.
Microshift Test Compatibility	✅ Passed	The pull request changes only CI YAML workflow configuration; it adds no Ginkgo tests or test code that could use MicroShift-unsupported APIs or features.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	The diff changes only YAML CI configuration and generated job definitions; it adds or modifies no Ginkgo e2e tests, so SNO compatibility review is not applicable.
Topology-Aware Scheduling Compatibility	✅ Passed	The PR changes only CI-operator test and generated Prow job YAML; it adds no deployment manifests, operator code, controllers, or workload scheduling constraints.
Ote Binary Stdout Contract	✅ Passed	The commit changes only 10 YAML configuration/job files; no OTE source, suite setup, logging, or stdout-writing code is added.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	The PR changes only CI YAML and generated job references; it adds no Ginkgo tests, IP parsing, hardcoded IPv4 values, URLs, or external connectivity code.
No-Weak-Crypto	✅ Passed	The 10 changed YAML files only update Azure conformance job metadata; the added-line scan found no MD5, SHA1, DES, RC4, Blowfish, ECB, crypto, or secret-comparison usage.
Container-Privileges	✅ Passed	The pull request changes only CI YAML and generated Prow job metadata; no privilege, host namespace, SYS_ADMIN, privilege-escalation, or explicit root settings were added.
No-Sensitive-Data-In-Logs	✅ Passed	The commit only changes CI job/configuration names, profiles, and test arguments; added-line scanning found no logging code or passwords, tokens, keys, PII, hostnames, or customer data.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

qinqon · 2026-08-06T11:55:28Z

/pj-rehearse

openshift-merge-bot · 2026-08-06T11:55:31Z

@qinqon: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

qinqon · 2026-08-06T11:56:15Z

/cc @orenc1

openshift-ci · 2026-08-06T11:56:54Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: qinqon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~ci-operator/config/openshift/cloud-provider-kubevirt/OWNERS~~ [qinqon]
~~ci-operator/jobs/openshift/cloud-provider-kubevirt/OWNERS~~ [qinqon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-merge-bot · 2026-08-06T11:58:04Z

[REHEARSALNOTIFIER]
@qinqon: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name	Repo	Type	Reason
pull-ci-openshift-cloud-provider-kubevirt-release-4.18-hypershift-kubevirt-conformance-azure	openshift/cloud-provider-kubevirt	presubmit	Presubmit changed
pull-ci-openshift-cloud-provider-kubevirt-release-4.19-hypershift-kubevirt-conformance-azure	openshift/cloud-provider-kubevirt	presubmit	Presubmit changed
pull-ci-openshift-cloud-provider-kubevirt-release-4.20-hypershift-kubevirt-conformance-azure	openshift/cloud-provider-kubevirt	presubmit	Presubmit changed
pull-ci-openshift-cloud-provider-kubevirt-release-4.21-hypershift-kubevirt-conformance-azure	openshift/cloud-provider-kubevirt	presubmit	Presubmit changed
pull-ci-openshift-cloud-provider-kubevirt-release-4.22-hypershift-kubevirt-conformance-azure	openshift/cloud-provider-kubevirt	presubmit	Presubmit changed

Prior to this PR being merged, you will need to either run and acknowledge or opt to skip these rehearsals.

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

qinqon · 2026-08-06T16:31:18Z

/pj-rehearse pull-ci-openshift-cloud-provider-kubevirt-release-4.20-hypershift-kubevirt-conformance-azure

openshift-merge-bot · 2026-08-06T16:31:22Z

@qinqon: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

openshift-ci · 2026-08-06T19:40:56Z

@qinqon: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci Bot requested a review from orenc1 August 6, 2026 11:56

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 6, 2026

coderabbitai Bot mentioned this pull request Aug 6, 2026

CNTRLPLANE-4010: add Azure AKS external OIDC periodic and presubmit jobs for 4.22 and 5.0 #83045

Open

4 tasks

Navigation Menu

Uh oh!

cloud-provider-kubevirt: move release-4.18..4.22 conformance jobs to Azure - #83031

cloud-provider-kubevirt: move release-4.18..4.22 conformance jobs to Azure#83031
qinqon wants to merge 1 commit into
openshift:mainfrom
qinqon:kubevirt-cpk-stable-azure-conformance

qinqon commented Aug 6, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

openshift-ci Bot commented Aug 6, 2026

Current state

What this changes

Test selection

Expected outcome

Notes for reviewers

Uh oh!

coderabbitai Bot commented Aug 6, 2026 •

edited

Loading

Uh oh!

qinqon commented Aug 6, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 6, 2026

Uh oh!

qinqon commented Aug 6, 2026

Uh oh!

openshift-ci Bot commented Aug 6, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 6, 2026

Uh oh!

qinqon commented Aug 6, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 6, 2026

Uh oh!

openshift-ci Bot commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

qinqon commented Aug 6, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Current state

What this changes

Test selection

Expected outcome

Notes for reviewers

Summary by CodeRabbit

Uh oh!

openshift-ci Bot commented Aug 6, 2026

Current state

What this changes

Test selection

Expected outcome

Notes for reviewers

Uh oh!

coderabbitai Bot commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

qinqon commented Aug 6, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 6, 2026

Uh oh!

qinqon commented Aug 6, 2026

Uh oh!

openshift-ci Bot commented Aug 6, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 6, 2026

Uh oh!

qinqon commented Aug 6, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 6, 2026

Uh oh!

openshift-ci Bot commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

qinqon commented Aug 6, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Aug 6, 2026 •

edited

Loading