Skip to content

OCPBUGS-63219: Support NLB protocol to configure proxy protocol and client IP preservation - #1426

Merged
openshift-merge-bot[bot] merged 7 commits into
openshift:masterfrom
gcs278:OCPBUGS-63219-proxy-protocol
Jul 31, 2026
Merged

OCPBUGS-63219: Support NLB protocol to configure proxy protocol and client IP preservation#1426
openshift-merge-bot[bot] merged 7 commits into
openshift:masterfrom
gcs278:OCPBUGS-63219-proxy-protocol

Conversation

@gcs278

@gcs278 gcs278 commented Apr 29, 2026

Copy link
Copy Markdown
Contributor

Summary

AWS NLBs have preserve_client_ip.enabled=true by default on their target groups. This causes hairpin connection failures on internal NLBs: when a pod sends traffic through the NLB and it routes back to the same node, the return packet's destination matches the pod's own IP, breaking the connection. This impacts ROSA private deployments and any internal NLB setup where the client pod and router pod share a node.

This PR adds a protocol field to AWSNetworkLoadBalancerParameters with two values:

  • PROXY — disables native client IP preservation and enables PROXY protocol v2 (preserve_client_ip.enabled=false,proxy_protocol_v2.enabled=true). HAProxy parses PROXY protocol headers to obtain the original client IP. Hairpin connections work.
  • TCP — uses AWS's native client IP preservation (preserve_client_ip.enabled=true). This is the current behavior. Hairpin connections may fail on internal NLBs.

The field reuses the same name and type already used by four other endpoint publishing strategies (HostNetwork, NodePort, Private, IBM), providing a consistent API surface: regardless of strategy or platform, protocol: PROXY means "enable PROXY protocol on the router and configure the infrastructure to match."

What this PR does

  • Service annotation: When PROXY, sets service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=false,proxy_protocol_v2.enabled=true on the NLB Service. When TCP, sets preserve_client_ip.enabled=true,proxy_protocol_v2.enabled=false. When protocol is empty (pre-existing NLB), no annotation is set — the operator does not manage the annotation for these ICs.
  • Router Deployment: When PROXY, sets ROUTER_USE_PROXY_PROTOCOL=true on the router so HAProxy parses PROXY protocol headers. The existing IsProxyProtocolNeeded function is updated to return true for NLBs with PROXY protocol.
  • Proxy protocol annotation separation: The CLB proxy protocol annotation (aws-load-balancer-proxy-protocol: "*") is now only set for CLBs. NLBs use the target-group-attributes annotation instead. A switch statement in desiredLoadBalancerService routes to the right annotation based on effective LB type and protocol.
  • LB type transition safety: Uses getEffectiveAWSLoadBalancerType to read the LB type from the current service annotation (not status) during pending CLB↔NLB transitions, preventing annotation stomping on the still-live service.
  • Status sync: IngressStatusesEqual is updated to compare protocol, and updatePublishingStrategy syncs spec changes to status. It also handles two additional scenarios:
    • Cleared spec: When a user removes a previously-set protocol from spec, the operator re-applies the PROXY default (status being non-empty proves the field was previously managed).
    • CLB→NLB transition: When the status LB type transitions to NLB and no protocol is set, the operator defaults to PROXY (detected via previousStatusLBType). Pre-existing NLBs on upgrade are excluded because their status type was already NLB (isPreExistingNLB).
  • Defaulting: setDefaultProviderParameters defaults new NLB IngressControllers to PROXY, gated by isNewIngressController (ic.Status.EndpointPublishingStrategy == nil). This is more reliable than !alreadyAdmitted because alreadyAdmitted can flip on re-admission after a validation failure.
  • Auto-delete safety: When the auto-delete-load-balancer annotation is set, the desired service and proxy protocol determination are computed from status rather than the current service. This prevents NLB-specific annotations from being applied to a new CLB service during LB type transitions, which the CCM would reject.
  • Hairpin risk alert: Emits an informational NLBHairpinRisk alert for existing internal NLB IngressControllers that have protocol unset, nudging admins to set PROXY (fix hairpin) or TCP (acknowledge current behavior). The alert description includes an oc edit command with the IC name populated from {{ $labels.name }}.

Upgrade compatibility

  • Existing IngressControllers are not modified on upgrade. The isNewIngressController check (status not yet initialized) ensures the PROXY default is only applied to brand new IngressControllers. Re-admitted ICs are not affected because their status is already initialized.
  • Empty protocol in status means "not managed." When an existing NLB has no protocol set (the pre-upgrade state), IsProxyProtocolNeeded returns false, the desired service gets no target-group-attributes annotation, and the one-directional reconciliation in loadBalancerServiceChanged only adds or updates the annotation — never removes it. Any user-set annotation (e.g., the KCS hairpin workaround) is left untouched.
  • CLB→NLB transitions get PROXY defaulted. When an already-admitted IC changes from CLB to NLB without explicitly setting the protocol, updatePublishingStrategy detects the type transition via previousStatusLBType and defaults to PROXY. This does not fire on upgrade because the status type was already NLB.
  • Mutability: The field is mutable and applied in-place — changing it does not require deleting or recreating the load balancer Service. The CCM supports updating NLB target group attributes in-place. There is a brief window during transition where connection failures may occur as the NLB attribute change and router rollout happen independently.

Design

Dependencies

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 29, 2026
@openshift-ci

openshift-ci Bot commented Apr 29, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci-robot openshift-ci-robot added jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. labels Apr 29, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@gcs278: This pull request references Jira Issue OCPBUGS-63219, which is invalid:

  • expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

  • Add support for the new clientIPPreservationMode field on AWSNetworkLoadBalancerParameters to control how client IP addresses are preserved by NLBs
  • When set to ProxyProtocol, configures the NLB target group with preserve_client_ip.enabled=false and proxy_protocol_v2.enabled=true, and enables ROUTER_USE_PROXY_PROTOCOL=true on the router — this fixes hairpin connection failures on internal NLBs (OCPBUGS-63219)
  • When set to Native (or omitted on existing IngressControllers), preserves current behavior using AWS's native client IP preservation
  • New IngressControllers default to ProxyProtocol via controller-managed defaulting (not CRD default), so existing ICs are not modified on upgrade

Dependencies

Test plan

  • Unit tests for desiredLoadBalancerService with ProxyProtocol and Native modes
  • Unit tests for IsProxyProtocolNeeded with NLB ProxyProtocol/Native
  • Unit tests for defaulting behavior (new vs existing ICs)
  • E2E: New NLB defaults to ProxyProtocol, connectivity works
  • E2E: CLB → NLB with ProxyProtocol, correct annotations, connectivity works
  • E2E: CLB → NLB with Native, no proxy annotations, connectivity works
  • E2E: NLB → CLB, CLB proxy annotation restored, connectivity works
  • E2E: Upgrade test — existing NLB not modified after deploying new operator
  • E2E: Hairpin test — internal NLB with ProxyProtocol, curl from same node succeeds
  • E2E: Hairpin test — internal NLB with Native, curl from same node times out (confirms bug)

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Apr 29, 2026
@coderabbitai

coderabbitai Bot commented Apr 29, 2026

Copy link
Copy Markdown

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Adds clientIPPreservationMode to AWS NLB CRD schemas in spec and status. Controller logic now defaults, propagates, and compares AWS NLB protocol state, and service reconciliation uses NLB target-group attributes for proxy protocol v2 when needed. A new hairpin-risk metric and alert were added. Unit and E2E tests cover proxy protocol, TCP protocol, default behavior, and related status and annotation updates. go.mod updates several resolved dependency versions.

🚥 Pre-merge checks | ✅ 12 | ❌ 3

❌ Failed checks (3 warnings)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Microshift Test Compatibility ⚠️ Warning The new e2e tests manage IngressController and config APIs with no MicroShift skip/tag; MicroShift only serves Route and SCC. Add a MicroShift guard ([Skipped:MicroShift], [apigroup:...], or runtime skip) or rework the test to avoid unsupported OpenShift APIs.
Ipv6 And Disconnected Network Test Compatibility ⚠️ Warning The new AWS NLB e2e test depends on external LB connectivity and uses http://%s URL building, which is not IPv6-safe. Use net.JoinHostPort for LB URLs and add IPv6/disconnected guarding or adaptation if the connectivity check must stay.
✅ Passed checks (12 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed PASS: New test titles are static literals (e.g. TestAWSNLBProtocol, t.Run("parallel")); no generated names, timestamps, UUIDs, or IPs appear in titles.
Test Structure And Quality ✅ Passed New tests use per-case table-driven units and E2E helpers with cleanup/timeouts; no missing cleanup or indefinite waits were introduced.
Single Node Openshift (Sno) Test Compatibility ✅ Passed PASS: The new e2e tests only provision an IngressController, echo pod, and verify LB annotations/connectivity; no node-count, drain, or multi-node scheduling assumptions were found.
Topology-Aware Scheduling Compatibility ✅ Passed No new scheduling constraints were added; changes are limited to NLB annotations, metrics, CRD fields, and tests, with no affinity/nodeSelector/topology-spread logic.
Ote Binary Stdout Contract ✅ Passed No PR changes add stdout writes in process-level code; the changed files use only t.Run/t.Logf and controller logic, with no fmt.Print, init, TestMain, or suite hooks.
No-Weak-Crypto ✅ Passed No added diff lines or touched functions reference weak ciphers, custom crypto, or secret comparisons; changes are limited to NLB config/metrics.
Container-Privileges ✅ Passed No changed manifest or code adds privileged/hostPID/hostNetwork/hostIPC/SYS_ADMIN or allowPrivilegeEscalation:true; only existing test assertions matched.
No-Sensitive-Data-In-Logs ✅ Passed Reviewed affected files; no new logging of secrets/PII/internal hostnames was added. Existing logs only emit resource names and protocol states.
Title check ✅ Passed The title clearly matches the main change: adding NLB protocol support for proxy protocol and client IP preservation.
Description check ✅ Passed The description is directly related to the changeset and explains the new NLB protocol behavior and rollout impact.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@gcs278
gcs278 force-pushed the OCPBUGS-63219-proxy-protocol branch from 4f32483 to c3b1e27 Compare April 29, 2026 01:52
@gcs278
gcs278 force-pushed the OCPBUGS-63219-proxy-protocol branch 2 times, most recently from 7ca9715 to 6a46693 Compare April 29, 2026 02:03
@gcs278
gcs278 marked this pull request as ready for review April 29, 2026 02:11
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 29, 2026
@gcs278 gcs278 changed the title OCPBUGS-63219: Support clientIPPreservationMode for AWS NLB [WIP] OCPBUGS-63219: Support clientIPPreservationMode for AWS NLB Apr 29, 2026
@gcs278

gcs278 commented Apr 29, 2026

Copy link
Copy Markdown
Contributor Author

/jira refresh

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 29, 2026
@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Apr 29, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@gcs278: This pull request references Jira Issue OCPBUGS-63219, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @anuragthehatter

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (2)
manifests/00-custom-resource-definition-OKD.yaml (1)

2760-2763: Optional: clarify status wording.

Line 2760–2763 says “the user has no opinion,” which is spec-oriented language. For the status schema, consider wording that reflects observed/effective state instead.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@manifests/00-custom-resource-definition-OKD.yaml` around lines 2760 - 2763,
Update the status schema description text that currently reads “the user has no
opinion” to language that reflects observed/effective state — e.g., replace that
phrase with “not specified by the user; the platform may choose a default
(currently 'ProxyProtocol')” and ensure any other status-related descriptions
use present-tense, observed wording rather than spec-oriented phrasing so the
status describes the effective value rather than intent.
test/e2e/nlb_client_ip_preservation_test.go (1)

37-38: Prefer generated IC names to reduce collision risk in repeated runs.

Using fixed names can cause intermittent AlreadyExists failures in reruns or partially cleaned environments.

♻️ Suggested change
-	name := types.NamespacedName{Namespace: operatorNamespace, Name: "nlb-pp-test"}
+	name := types.NamespacedName{Namespace: operatorNamespace, Name: names.SimpleNameGenerator.GenerateName("nlb-pp-")}
...
-	name := types.NamespacedName{Namespace: operatorNamespace, Name: "nlb-default"}
+	name := types.NamespacedName{Namespace: operatorNamespace, Name: names.SimpleNameGenerator.GenerateName("nlb-default-")}

Also applies to: 125-126

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/e2e/nlb_client_ip_preservation_test.go` around lines 37 - 38, The test
currently uses a fixed NamespacedName (variable name :=
types.NamespacedName{Namespace: operatorNamespace, Name: "nlb-pp-test"}) and
derived domain, which risks AlreadyExists on reruns; change the creation to
generate a unique resource name (e.g., append a short random/UUID/timestamp
suffix) when constructing the types.NamespacedName and update the derived domain
assignment (domain := name.Name + "." + dnsConfig.Spec.BaseDomain) accordingly;
apply the same change to the other fixed NamespacedName usage referenced around
lines 125-126 so all test resources use generated unique names to avoid
collisions.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@go.mod`:
- Line 229: The go.mod contains a replace directive "replace
github.com/openshift/api => github.com/gcs278/api
v0.0.0-20260429000454-cff0427099ea" that introduces a fork without explanation;
either remove this replace to use the upstream github.com/openshift/api, or if
the fork is required keep the replace but add a nearby comment explaining why
the fork is necessary (reference the commit cff0427099ea and the specific
issue/bug it fixes), mark it as TODO with a target removal date/version, and
ensure the rationale is committed so reviewers know it’s intentional.

In `@test/e2e/nlb_client_ip_preservation_test.go`:
- Around line 169-171: The wait timeout in
waitForIngressControllerClientIPPreservationMode is set to 2 minutes causing
intermittent flakes; update the timeout argument in the
wait.PollUntilContextTimeout call from 2*time.Minute to 5*time.Minute to match
other readiness waits so reconciliation has more time to settle (reference
function name waitForIngressControllerClientIPPreservationMode and the
wait.PollUntilContextTimeout invocation).

---

Nitpick comments:
In `@manifests/00-custom-resource-definition-OKD.yaml`:
- Around line 2760-2763: Update the status schema description text that
currently reads “the user has no opinion” to language that reflects
observed/effective state — e.g., replace that phrase with “not specified by the
user; the platform may choose a default (currently 'ProxyProtocol')” and ensure
any other status-related descriptions use present-tense, observed wording rather
than spec-oriented phrasing so the status describes the effective value rather
than intent.

In `@test/e2e/nlb_client_ip_preservation_test.go`:
- Around line 37-38: The test currently uses a fixed NamespacedName (variable
name := types.NamespacedName{Namespace: operatorNamespace, Name: "nlb-pp-test"})
and derived domain, which risks AlreadyExists on reruns; change the creation to
generate a unique resource name (e.g., append a short random/UUID/timestamp
suffix) when constructing the types.NamespacedName and update the derived domain
assignment (domain := name.Name + "." + dnsConfig.Spec.BaseDomain) accordingly;
apply the same change to the other fixed NamespacedName usage referenced around
lines 125-126 so all test resources use generated unique names to avoid
collisions.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 7aaec5b4-af35-4500-903f-2af108380f15

📥 Commits

Reviewing files that changed from the base of the PR and between 961ac21 and 6a46693.

⛔ Files ignored due to path filters (49)
  • go.sum is excluded by !**/*.sum
  • vendor/github.com/openshift/api/.golangci.yaml is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/config/v1/types.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/config/v1/types_authentication.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/config/v1/types_infrastructure.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/config/v1/zz_generated.deepcopy.go is excluded by !**/vendor/**, !vendor/**, !**/zz_generated*
  • vendor/github.com/openshift/api/config/v1/zz_generated.featuregated-crd-manifests.yaml is excluded by !**/vendor/**, !vendor/**, !**/zz_generated*
  • vendor/github.com/openshift/api/config/v1/zz_generated.swagger_doc_generated.go is excluded by !**/vendor/**, !vendor/**, !**/zz_generated*
  • vendor/github.com/openshift/api/config/v1alpha1/types_cluster_monitoring.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/config/v1alpha1/zz_generated.deepcopy.go is excluded by !**/vendor/**, !vendor/**, !**/zz_generated*
  • vendor/github.com/openshift/api/config/v1alpha1/zz_generated.swagger_doc_generated.go is excluded by !**/vendor/**, !vendor/**, !**/zz_generated*
  • vendor/github.com/openshift/api/envtest-releases.yaml is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/etcd/install.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/etcd/v1/Makefile is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/etcd/v1/doc.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/etcd/v1/register.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/etcd/v1/types_pacemakercluster.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/etcd/v1/zz_generated.deepcopy.go is excluded by !**/vendor/**, !vendor/**, !**/zz_generated*
  • vendor/github.com/openshift/api/etcd/v1/zz_generated.featuregated-crd-manifests.yaml is excluded by !**/vendor/**, !vendor/**, !**/zz_generated*
  • vendor/github.com/openshift/api/etcd/v1/zz_generated.swagger_doc_generated.go is excluded by !**/vendor/**, !vendor/**, !**/zz_generated*
  • vendor/github.com/openshift/api/etcd/v1alpha1/types_pacemakercluster.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/etcd/v1alpha1/zz_generated.swagger_doc_generated.go is excluded by !**/vendor/**, !vendor/**, !**/zz_generated*
  • vendor/github.com/openshift/api/features.md is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/features/features.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/machine/v1beta1/types_machineset.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/machine/v1beta1/zz_generated.swagger_doc_generated.go is excluded by !**/vendor/**, !vendor/**, !**/zz_generated*
  • vendor/github.com/openshift/api/operator/v1/types_ingress.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/operator/v1/zz_generated.crd-manifests/0000_20_kube-apiserver_01_kubeapiservers-Default.crd.yaml is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/operator/v1/zz_generated.crd-manifests/0000_20_kube-apiserver_01_kubeapiservers-DevPreviewNoUpgrade.crd.yaml is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/operator/v1/zz_generated.crd-manifests/0000_20_kube-apiserver_01_kubeapiservers-OKD.crd.yaml is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/operator/v1/zz_generated.crd-manifests/0000_20_kube-apiserver_01_kubeapiservers-TechPreviewNoUpgrade.crd.yaml is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/operator/v1/zz_generated.crd-manifests/0000_20_kube-apiserver_01_kubeapiservers.crd.yaml is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/operator/v1/zz_generated.crd-manifests/0000_50_ingress_00_ingresscontrollers-CustomNoUpgrade.crd.yaml is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/operator/v1/zz_generated.crd-manifests/0000_50_ingress_00_ingresscontrollers-Default.crd.yaml is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/operator/v1/zz_generated.crd-manifests/0000_50_ingress_00_ingresscontrollers-DevPreviewNoUpgrade.crd.yaml is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/operator/v1/zz_generated.crd-manifests/0000_50_ingress_00_ingresscontrollers-OKD.crd.yaml is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/operator/v1/zz_generated.crd-manifests/0000_50_ingress_00_ingresscontrollers-TechPreviewNoUpgrade.crd.yaml is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/operator/v1/zz_generated.swagger_doc_generated.go is excluded by !**/vendor/**, !vendor/**, !**/zz_generated*
  • vendor/github.com/openshift/api/operator/v1alpha1/types_clusterapi.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/operator/v1alpha1/zz_generated.deepcopy.go is excluded by !**/vendor/**, !vendor/**, !**/zz_generated*
  • vendor/github.com/openshift/api/operator/v1alpha1/zz_generated.swagger_doc_generated.go is excluded by !**/vendor/**, !vendor/**, !**/zz_generated*
  • vendor/github.com/openshift/api/quota/v1/generated.proto is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/quota/v1/types.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/quota/v1/zz_generated.featuregated-crd-manifests.yaml is excluded by !**/vendor/**, !vendor/**, !**/zz_generated*
  • vendor/github.com/openshift/api/security/v1/generated.proto is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/security/v1/types.go is excluded by !**/vendor/**, !vendor/**
  • vendor/github.com/openshift/api/security/v1/zz_generated.featuregated-crd-manifests.yaml is excluded by !**/vendor/**, !vendor/**, !**/zz_generated*
  • vendor/github.com/openshift/api/security/v1/zz_generated.swagger_doc_generated.go is excluded by !**/vendor/**, !vendor/**, !**/zz_generated*
  • vendor/modules.txt is excluded by !**/vendor/**, !vendor/**
📒 Files selected for processing (13)
  • go.mod
  • manifests/00-custom-resource-definition-CustomNoUpgrade.yaml
  • manifests/00-custom-resource-definition-DevPreviewNoUpgrade.yaml
  • manifests/00-custom-resource-definition-OKD.yaml
  • manifests/00-custom-resource-definition-TechPreviewNoUpgrade.yaml
  • manifests/00-custom-resource-definition.yaml
  • pkg/operator/controller/ingress/controller.go
  • pkg/operator/controller/ingress/controller_test.go
  • pkg/operator/controller/ingress/load_balancer_service.go
  • pkg/operator/controller/ingress/load_balancer_service_test.go
  • pkg/operator/controller/ingress/status.go
  • test/e2e/all_test.go
  • test/e2e/nlb_client_ip_preservation_test.go

Comment thread go.mod Outdated
Comment thread test/e2e/nlb_client_ip_preservation_test.go Outdated
@gcs278

gcs278 commented Apr 29, 2026

Copy link
Copy Markdown
Contributor Author

/retest

@gcs278
gcs278 force-pushed the OCPBUGS-63219-proxy-protocol branch from 6a46693 to dbb29d0 Compare April 29, 2026 14:14

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
test/e2e/nlb_client_ip_preservation_test.go (1)

37-38: Use generated IngressController names to reduce rerun collision flakes.

Since these tests run in parallel, fixed names ("nlb-pp-test", "nlb-default") can conflict with leftovers from interrupted runs. Prefer generated names.

♻️ Suggested change
- name := types.NamespacedName{Namespace: operatorNamespace, Name: "nlb-pp-test"}
+ name := types.NamespacedName{
+   Namespace: operatorNamespace,
+   Name:      names.SimpleNameGenerator.GenerateName("nlb-pp-test-"),
+ }
  domain := name.Name + "." + dnsConfig.Spec.BaseDomain
- name := types.NamespacedName{Namespace: operatorNamespace, Name: "nlb-default"}
+ name := types.NamespacedName{
+   Namespace: operatorNamespace,
+   Name:      names.SimpleNameGenerator.GenerateName("nlb-default-"),
+ }
  domain := name.Name + "." + dnsConfig.Spec.BaseDomain

Also applies to: 125-126

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/e2e/nlb_client_ip_preservation_test.go` around lines 37 - 38, Replace
the fixed IngressController names with generated, unique names to avoid parallel
test collisions: instead of using the literal "nlb-pp-test" when constructing
the NamespacedName stored in the variable name (and building domain from
name.Name), generate a unique name (e.g., using a random/suffix or Kubernetes
GenerateName pattern) and use that value for name.Name and domain; do the same
replacement for the other hardcoded "nlb-default" occurrence (lines around where
that literal is used). Update any assertions or cleanup that reference those
literals to use the generated name variables (name and domain) so tests don't
collide across runs.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@test/e2e/nlb_client_ip_preservation_test.go`:
- Around line 37-38: Replace the fixed IngressController names with generated,
unique names to avoid parallel test collisions: instead of using the literal
"nlb-pp-test" when constructing the NamespacedName stored in the variable name
(and building domain from name.Name), generate a unique name (e.g., using a
random/suffix or Kubernetes GenerateName pattern) and use that value for
name.Name and domain; do the same replacement for the other hardcoded
"nlb-default" occurrence (lines around where that literal is used). Update any
assertions or cleanup that reference those literals to use the generated name
variables (name and domain) so tests don't collide across runs.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: d9f49a9a-0e30-4b40-b195-94d238657d59

📥 Commits

Reviewing files that changed from the base of the PR and between 6a46693 and dbb29d0.

📒 Files selected for processing (2)
  • test/e2e/all_test.go
  • test/e2e/nlb_client_ip_preservation_test.go

@jcmoraisjr

Copy link
Copy Markdown
Member

/assign

@gcs278
gcs278 force-pushed the OCPBUGS-63219-proxy-protocol branch from dbb29d0 to 2c0d8a4 Compare May 28, 2026 21:06
@coderabbitai

coderabbitai Bot commented May 28, 2026

Copy link
Copy Markdown

Actionable comments posted: 0

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Tests from second stage were triggered manually. Pipeline can be controlled only manually, until HEAD changes. Use command to trigger second stage.

@openshift-ci

openshift-ci Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

@gcs278: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-operator-techpreview 88c59fc link false /test e2e-aws-operator-techpreview

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@gcs278

gcs278 commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

/pipeline required

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-gatewayapi-conformance
/test e2e-aws-operator
/test e2e-aws-ovn
/test e2e-aws-ovn-hypershift-conformance
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-aws-ovn-upgrade
/test e2e-azure-operator
/test e2e-gcp-operator
/test e2e-hypershift
/test e2e-vsphere-static-metallb-operator-gwapi
/test e2e-vsphere-static-metallb-operator-gwapi-techpreview

gcs278 and others added 6 commits July 30, 2026 13:04
…safety

Add TestAWSNLBProtocol and TestAWSNLBDefaultProtocol to verify NLB
protocol annotation, router PROXY protocol env var, connectivity,
and defaulting behavior for new IngressControllers.

Add TestAWSNLBUpgradeAnnotationPreservation to simulate upgrading a
pre-existing NLB IngressController and verify the operator does not
stomp user-set target-group-attributes annotations.

Add TestAWSLBTypeTransitionSafety to verify that CLB/NLB type
transitions do not break traffic or stomp proxy protocol annotations
while the service is in a pending transition state.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…lers

Add an informational Prometheus alert that fires when an IngressController
uses an internal AWS NLB with an empty NLB protocol in status. An empty
NLB protocol means the operator is not managing the target-group-
attributes annotation for this IC, so it may be using AWS's native
client IP preservation and could be affected by hairpin connection
failures. The alert nudges admins to set protocol to PROXY (to fix
hairpin) or TCP (to acknowledge current behavior and silence the alert).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… for NLB protocol defaulting

Replace the !alreadyAdmitted gate on NLB protocol defaulting with
isNewIngressController (ic.Status.EndpointPublishingStrategy == nil).
The alreadyAdmitted flag can flip when an IC is re-admitted after a
validation failure, which would incorrectly re-default the protocol
to PROXY for existing NLBs. Checking whether status has been
initialized is a reliable signal for truly new IngressControllers.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Refactor the proxy protocol annotation switch to use effectiveLBType
as the outer switch with proxyNeeded for value selection, guarded by
len(nlbProtocol) > 0 for upgrade safety. Use t.Context() instead of
context.TODO() in e2e tests.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add upgrade context, traffic disruption warning, and TCP trade-off
explanation to the alert description.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ring auto-delete

When auto-delete is set, the service will be deleted and recreated in
the same reconcile. Pass nil for currentService so that
desiredLoadBalancerService and IsProxyProtocolNeeded compute state from
status (the intended state) rather than from the soon-to-be-deleted
service. This prevents NLB-specific annotations from being set on a
new CLB service, which the CCM rejects with "target group attributes
annotation is only supported for NLB."

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@gcs278

gcs278 commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

@jcmoraisjr Sorry for the extra review round: the proxy annotation switch refactor you suggested actually exposed a bug in the auto-delete workflow.

When auto-delete is set and the LB type changes from NLB→CLB, desiredLoadBalancerService was computed using the old NLB service. This caused NLB-specific annotations (target-group-attributes) to bleed onto the new CLB service. The same staleness affected the router's proxy protocol determination, but the difference is that a wrong env var on the router self-heals on the next reconcile — the CCM does not:

"message": "The service-controller component is reporting SyncLoadBalancerFailed events like: 
Error syncing load balancer: failed to ensure load balancer: target group attributes annotation 
is only supported for NLB"

The fix: when auto-delete is set, pass nil for currentService so the desired service and proxy protocol determination are computed from status (the intended state) rather than from the soon-to-be-deleted service.

Also reduced TestAWSLBTypeTransitionSafety from 6 to 4 test cases — the dropped cases were testing slightly different flavors of the same code path, and creating that many NLBs in parallel was adding unnecessary load that concerns me regarding our E2E quotas.

@gcs278

gcs278 commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

/pipeline required

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-gatewayapi-conformance
/test e2e-aws-operator
/test e2e-aws-ovn
/test e2e-aws-ovn-hypershift-conformance
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-aws-ovn-upgrade
/test e2e-azure-operator
/test e2e-gcp-operator
/test e2e-hypershift
/test e2e-vsphere-static-metallb-operator-gwapi
/test e2e-vsphere-static-metallb-operator-gwapi-techpreview

@jcmoraisjr

Copy link
Copy Markdown
Member

I don't know whether I'm glad to help the finding or sorry to have added the noise 🙂

/lgtm

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Tests from second stage were triggered manually. Pipeline can be controlled only manually, until HEAD changes. Use command to trigger second stage.

@gcs278

gcs278 commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

/payload-job periodic-ci-openshift-release-main-ci-5.0-e2e-aws-ovn-upgrade

@openshift-ci

openshift-ci Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

@gcs278: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-ci-5.0-e2e-aws-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/1f49a5d0-8c3d-11f1-835b-bfe75395c83d-0

@gcs278

gcs278 commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

install failure
/test e2e-gcp-operator

@gcs278

gcs278 commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

NAME TestAll/parallel/TestContainerLoggingMinLength operator_test.go:2887: failed to create network policy syslog-netpol: networkpolicies.networking.k8s.io "syslog-netpol" already exists known bug:
/test e2e-aws-operator

@gcs278

gcs278 commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

this job is known to be flaky today:
--- FAIL: TestAutoscaling/ValidateHostedCluster/EnsureNoCrashingPods (0.30s)

/test hypershfit-e2e-aks

@gcs278

gcs278 commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

/test hypershift-e2e-aks

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@gcs278: This pull request references Jira Issue OCPBUGS-63219, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @anuragthehatter

Details

In response to this:

Summary

AWS NLBs have preserve_client_ip.enabled=true by default on their target groups. This causes hairpin connection failures on internal NLBs: when a pod sends traffic through the NLB and it routes back to the same node, the return packet's destination matches the pod's own IP, breaking the connection. This impacts ROSA private deployments and any internal NLB setup where the client pod and router pod share a node.

This PR adds a protocol field to AWSNetworkLoadBalancerParameters with two values:

  • PROXY — disables native client IP preservation and enables PROXY protocol v2 (preserve_client_ip.enabled=false,proxy_protocol_v2.enabled=true). HAProxy parses PROXY protocol headers to obtain the original client IP. Hairpin connections work.
  • TCP — uses AWS's native client IP preservation (preserve_client_ip.enabled=true). This is the current behavior. Hairpin connections may fail on internal NLBs.

The field reuses the same name and type already used by four other endpoint publishing strategies (HostNetwork, NodePort, Private, IBM), providing a consistent API surface: regardless of strategy or platform, protocol: PROXY means "enable PROXY protocol on the router and configure the infrastructure to match."

What this PR does

  • Service annotation: When PROXY, sets service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=false,proxy_protocol_v2.enabled=true on the NLB Service. When TCP, sets preserve_client_ip.enabled=true,proxy_protocol_v2.enabled=false. When protocol is empty (pre-existing NLB), no annotation is set — the operator does not manage the annotation for these ICs.
  • Router Deployment: When PROXY, sets ROUTER_USE_PROXY_PROTOCOL=true on the router so HAProxy parses PROXY protocol headers. The existing IsProxyProtocolNeeded function is updated to return true for NLBs with PROXY protocol.
  • Proxy protocol annotation separation: The CLB proxy protocol annotation (aws-load-balancer-proxy-protocol: "*") is now only set for CLBs. NLBs use the target-group-attributes annotation instead. A switch statement in desiredLoadBalancerService routes to the right annotation based on effective LB type and protocol.
  • LB type transition safety: Uses getEffectiveAWSLoadBalancerType to read the LB type from the current service annotation (not status) during pending CLB↔NLB transitions, preventing annotation stomping on the still-live service.
  • Status sync: IngressStatusesEqual is updated to compare protocol, and updatePublishingStrategy syncs spec changes to status. It also handles two additional scenarios:
  • Cleared spec: When a user removes a previously-set protocol from spec, the operator re-applies the PROXY default (status being non-empty proves the field was previously managed).
  • CLB→NLB transition: When the status LB type transitions to NLB and no protocol is set, the operator defaults to PROXY (detected via previousStatusLBType). Pre-existing NLBs on upgrade are excluded because their status type was already NLB (isPreExistingNLB).
  • Defaulting: setDefaultProviderParameters defaults new NLB IngressControllers to PROXY, gated by isNewIngressController (ic.Status.EndpointPublishingStrategy == nil). This is more reliable than !alreadyAdmitted because alreadyAdmitted can flip on re-admission after a validation failure.
  • Auto-delete safety: When the auto-delete-load-balancer annotation is set, the desired service and proxy protocol determination are computed from status rather than the current service. This prevents NLB-specific annotations from being applied to a new CLB service during LB type transitions, which the CCM would reject.
  • Hairpin risk alert: Emits an informational NLBHairpinRisk alert for existing internal NLB IngressControllers that have protocol unset, nudging admins to set PROXY (fix hairpin) or TCP (acknowledge current behavior). The alert description includes an oc edit command with the IC name populated from {{ $labels.name }}.

Upgrade compatibility

  • Existing IngressControllers are not modified on upgrade. The isNewIngressController check (status not yet initialized) ensures the PROXY default is only applied to brand new IngressControllers. Re-admitted ICs are not affected because their status is already initialized.
  • Empty protocol in status means "not managed." When an existing NLB has no protocol set (the pre-upgrade state), IsProxyProtocolNeeded returns false, the desired service gets no target-group-attributes annotation, and the one-directional reconciliation in loadBalancerServiceChanged only adds or updates the annotation — never removes it. Any user-set annotation (e.g., the KCS hairpin workaround) is left untouched.
  • CLB→NLB transitions get PROXY defaulted. When an already-admitted IC changes from CLB to NLB without explicitly setting the protocol, updatePublishingStrategy detects the type transition via previousStatusLBType and defaults to PROXY. This does not fire on upgrade because the status type was already NLB.
  • Mutability: The field is mutable and applied in-place — changing it does not require deleting or recreating the load balancer Service. The CCM supports updating NLB target group attributes in-place. There is a brief window during transition where connection failures may occur as the NLB attribute change and router rollout happen independently.

Design

Dependencies

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@gcs278

gcs278 commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

Copying @melvinjoseph86's verified:
/verified later @mjoseph

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@gcs278: This PR has been marked to be verified later by @mjoseph.

Details

In response to this:

Copying @melvinjoseph86's verified:
/verified later @mjoseph

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@gcs278

gcs278 commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

* could not run steps: step e2e-gcp-operator failed: "e2e-gcp-operator" post steps failed: "e2e-gcp-operator" pod "e2e-gcp-operator-gather-gcp-console" failed: could not watch pod: the pod ci-op-j897g344/e2e-gcp-operator-gather-gcp-console failed after 1m2s (failed containers: test): ContainerFailed one or more containers exited

/test e2e-gcp-operator

@gcs278

gcs278 commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

{ failed to acquire lease for "hypershift-aks-quota-slice": resources not found}:
/test hypershift-e2e-aks

@gcs278

gcs278 commented Jul 31, 2026

Copy link
Copy Markdown
Contributor Author

I think we need to proceed merging with this one as is. It's not a perfect solution, but it is a pragmatic one, this fix is in high demand. In the future, I may try to add some sort of one-off migration logic so that we don't have these subtle status behaviors.

I also would like to add E2E tests for the next metric/alert, but since we are out of time, we can merge as is, and I'll provide a follow up.

/unhold

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@gcs278: Jira Issue OCPBUGS-63219: Some pull requests linked via external trackers have merged:

The following pull request, linked via external tracker, has not merged:

All associated pull requests must be merged or unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with /jira refresh.

Jira Issue OCPBUGS-63219 has not been moved to the MODIFIED state.

This PR is marked as verified-later. Jira issue(s) in the title of this PR will require post-merge verification. After testing, it must be manually moved to the VERIFIED state.

Details

In response to this:

Summary

AWS NLBs have preserve_client_ip.enabled=true by default on their target groups. This causes hairpin connection failures on internal NLBs: when a pod sends traffic through the NLB and it routes back to the same node, the return packet's destination matches the pod's own IP, breaking the connection. This impacts ROSA private deployments and any internal NLB setup where the client pod and router pod share a node.

This PR adds a protocol field to AWSNetworkLoadBalancerParameters with two values:

  • PROXY — disables native client IP preservation and enables PROXY protocol v2 (preserve_client_ip.enabled=false,proxy_protocol_v2.enabled=true). HAProxy parses PROXY protocol headers to obtain the original client IP. Hairpin connections work.
  • TCP — uses AWS's native client IP preservation (preserve_client_ip.enabled=true). This is the current behavior. Hairpin connections may fail on internal NLBs.

The field reuses the same name and type already used by four other endpoint publishing strategies (HostNetwork, NodePort, Private, IBM), providing a consistent API surface: regardless of strategy or platform, protocol: PROXY means "enable PROXY protocol on the router and configure the infrastructure to match."

What this PR does

  • Service annotation: When PROXY, sets service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=false,proxy_protocol_v2.enabled=true on the NLB Service. When TCP, sets preserve_client_ip.enabled=true,proxy_protocol_v2.enabled=false. When protocol is empty (pre-existing NLB), no annotation is set — the operator does not manage the annotation for these ICs.
  • Router Deployment: When PROXY, sets ROUTER_USE_PROXY_PROTOCOL=true on the router so HAProxy parses PROXY protocol headers. The existing IsProxyProtocolNeeded function is updated to return true for NLBs with PROXY protocol.
  • Proxy protocol annotation separation: The CLB proxy protocol annotation (aws-load-balancer-proxy-protocol: "*") is now only set for CLBs. NLBs use the target-group-attributes annotation instead. A switch statement in desiredLoadBalancerService routes to the right annotation based on effective LB type and protocol.
  • LB type transition safety: Uses getEffectiveAWSLoadBalancerType to read the LB type from the current service annotation (not status) during pending CLB↔NLB transitions, preventing annotation stomping on the still-live service.
  • Status sync: IngressStatusesEqual is updated to compare protocol, and updatePublishingStrategy syncs spec changes to status. It also handles two additional scenarios:
  • Cleared spec: When a user removes a previously-set protocol from spec, the operator re-applies the PROXY default (status being non-empty proves the field was previously managed).
  • CLB→NLB transition: When the status LB type transitions to NLB and no protocol is set, the operator defaults to PROXY (detected via previousStatusLBType). Pre-existing NLBs on upgrade are excluded because their status type was already NLB (isPreExistingNLB).
  • Defaulting: setDefaultProviderParameters defaults new NLB IngressControllers to PROXY, gated by isNewIngressController (ic.Status.EndpointPublishingStrategy == nil). This is more reliable than !alreadyAdmitted because alreadyAdmitted can flip on re-admission after a validation failure.
  • Auto-delete safety: When the auto-delete-load-balancer annotation is set, the desired service and proxy protocol determination are computed from status rather than the current service. This prevents NLB-specific annotations from being applied to a new CLB service during LB type transitions, which the CCM would reject.
  • Hairpin risk alert: Emits an informational NLBHairpinRisk alert for existing internal NLB IngressControllers that have protocol unset, nudging admins to set PROXY (fix hairpin) or TCP (acknowledge current behavior). The alert description includes an oc edit command with the IC name populated from {{ $labels.name }}.

Upgrade compatibility

  • Existing IngressControllers are not modified on upgrade. The isNewIngressController check (status not yet initialized) ensures the PROXY default is only applied to brand new IngressControllers. Re-admitted ICs are not affected because their status is already initialized.
  • Empty protocol in status means "not managed." When an existing NLB has no protocol set (the pre-upgrade state), IsProxyProtocolNeeded returns false, the desired service gets no target-group-attributes annotation, and the one-directional reconciliation in loadBalancerServiceChanged only adds or updates the annotation — never removes it. Any user-set annotation (e.g., the KCS hairpin workaround) is left untouched.
  • CLB→NLB transitions get PROXY defaulted. When an already-admitted IC changes from CLB to NLB without explicitly setting the protocol, updatePublishingStrategy detects the type transition via previousStatusLBType and defaults to PROXY. This does not fire on upgrade because the status type was already NLB.
  • Mutability: The field is mutable and applied in-place — changing it does not require deleting or recreating the load balancer Service. The CCM supports updating NLB target group attributes in-place. There is a brief window during transition where connection failures may occur as the NLB attribute change and router rollout happen independently.

Design

Dependencies

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-merge-robot

Copy link
Copy Markdown
Contributor

Fix included in release 5.0.0-0.nightly-2026-08-04-172547

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria verified-later

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants