Azure: cache VNet validation result in validateConfigAndClusterCapabilities to avoid per-reconcile ARM 429 throttling

[miquelsi]

Problem

The validateConfigAndClusterCapabilities function in
control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go
calls the Azure ARM API to GET .../Microsoft.Network/virtualNetworks/<name> on every
reconcile loop to verify that the VNet, NSG, and managed resource group locations match.
There is no caching, no respect for the Retry-After response header, and no rate limiting
of this call.

Under load — for example, when many clusters are being provisioned concurrently on the same
Azure subscription — this generates a high frequency of Microsoft.Network GET calls. Azure
responds with:

RESPONSE 429: Too Many Requests
Code: RetryableErrorDueToTooManyCalls
"Subscription <id> was used to perform too many calls within last 5 minutes.
The number of calls exceeds Microsoft.Network throttling limit."

Impact

When the 429 is returned, validateConfigAndClusterCapabilities returns an error, which
causes the reconciler to set:

ValidHostedControlPlaneConfiguration = False
Reason: InsufficientClusterCapabilities

The reconciler then hard-gates on this condition (returns early with no requeue) until
the next watch event triggers a new reconcile — which immediately calls Azure ARM again and
gets another 429. This creates a tight retry loop that holds
ValidHostedControlPlaneConfiguration = False for as long as the subscription remains
throttled (observed: 12+ minutes in production).

During this entire window, the full HCP reconciliation is blocked:

• CNO cannot configure OVN-Kubernetes networking
• Worker nodes can't initialize (no CNI → Node condition Ready = False)
• Node pool creation operations time out and are marked Failed

In a CI environment with ~400 concurrent clusters observed in a single 42-minute window,
31% of clusters received 429 errors on the VNet GET call.

Root cause

The VNet location is immutable for the lifetime of a cluster — it never changes. There
is no reason to call Azure ARM on every reconcile loop. The combination of:

1. No caching of the validation result
2. No respect for the Retry-After header in the 429 response
3. A hard reconcile gate when the condition is False (causing rapid retries)

…means a single throttle event cascades into a multi-minute installation stall.

Relevant code: hostedcontrolplane_controller.go

• Lines ~619–625: condition is set
• Line ~3264: GetVnetInfoFromVnetID call
• Lines ~710–715: hard gate that blocks reconciliation when condition is False

Proposed fix

Two complementary changes:

1. Cache the validation result

Since the VNet location is immutable, skip the Azure ARM call if
ValidHostedControlPlaneConfiguration is already True. Only re-validate when relevant
spec fields change (e.g. VnetID, SubnetID, NetworkSecurityGroupID) or when the
condition has never been evaluated (status is Unknown).

// Skip Azure ARM call if config was already validated and no relevant spec fields changed.
// VNet location is immutable for the lifetime of a cluster.
validConfig := meta.FindStatusCondition(hcp.Status.Conditions, string(hyperv1.ValidHostedControlPlaneConfiguration))
if validConfig != nil && validConfig.Status == metav1.ConditionTrue {
    return nil
}

2. Respect Retry-After on 429

When GetVnetInfoFromVnetID receives a 429, extract the Retry-After header and return
ctrl.Result{RequeueAfter: retryAfterDuration} instead of immediately re-entering the
reconcile loop. This prevents the stampeding-herd retry pattern that holds the condition
False for the entire throttle window.

Evidence from logs

 ### Condition blocked for 12 minutes (12:52 → 13:04 UTC)
 Set HostedControlPlane conditions:
 [{ValidHostedControlPlaneConfiguration False 1 2026-07-21 12:51:51 +0000 UTC InsufficientClusterCapabilities failed to get vnet info to verify its location: failed to get virtual network: GET .../Microsoft.Network/virtualNetworks/customer-vnet RESPONSE 429: Too Many Requests}]

Condition only resolved at 13:04:37 — after the node pool operation deadline (13:14) was already lost

 Set HostedControlPlane conditions:
 [{ValidHostedControlPlaneConfiguration True 1 2026-07-21 13:04:37 +0000 UTC ...}]