feat: add HCP pod churn ratio panel to gather-observability by geoberle · Pull Request #6836 · Azure/ARO-HCP

Gerd Oberlechner (geoberle) · 2026-09-04T16:36:16Z

Summary

Adds a "HCP Control Plane Pod Churn Ratio" query to the existing "MC AKS Metrics" panel in test/cmd/aro-hcp-tests/gather-observability/queries.yaml.
Reports distinct pods created per Deployment (HostedControlPlane namespaces only) divided by desired replica count, in 1-minute tumbling windows. ~1 = healthy; higher means pods were repeatedly replaced rather than just scaled up.
Useful for spotting node instability / crash-loop-driven pod replacement during a HCP's short (~30 min) lifetime, which tends to concentrate in the first few minutes after creation.
Reuses the existing namespace_workload_pod:kube_pod_owner:relabel recording rule (kube-prometheus-mixin, already deployed to management clusters) for the pod→Deployment join instead of re-deriving it.

Test plan

go test ./gather-observability/... -run TestLoadQueriesConfigEmbedded passes
make -C config materialize passes with no diff issues
Visually confirm the new chart renders correctly in a real gather-observability run (Spyglass artifact) once available on a PR/CI job

openshift-ci · 2026-09-04T16:36:24Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: geoberle

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~test/OWNERS~~ [geoberle]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copilot

🟡 Changes recommended

The new PromQL uses count_over_time in a way that counts scrape samples rather than distinct pods, making the “~1 = healthy” ratio incorrect and scrape-interval-dependent.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR extends the gather-observability PromQL query set by adding a new panel intended to highlight excessive HostedControlPlane pod replacement (“churn”) early in an HCP cluster’s lifecycle.

Changes:

Adds a new “HCP Control Plane Pod Churn Ratio” PromQL chart based on pod→Deployment ownership and desired replica count.
Sets a peak threshold (minPeakThreshold: 2) for flagging elevated churn levels.

File summaries

File	Description
`test/cmd/aro-hcp-tests/gather-observability/queries.yaml`	Adds a new churn-ratio chart definition for HCP control plane pods.

Review details

Suppressed comments (1)

test/cmd/aro-hcp-tests/gather-observability/queries.yaml:453

count_over_time(...) counts the number of samples scraped for each pod timeseries in the 1m range (typically ~2 if scrape interval is 30s), not the number of distinct pods. That makes the ratio baseline depend on scrape interval and will tend to be ~scrapes-per-minute even when there is no churn, which contradicts the intended "~1 = healthy" meaning and will also interact badly with minPeakThreshold: 2. Use max_over_time(...) to collapse each pod series to a single 1/0 presence value, then count those series to approximate distinct pods seen in the window.

      count by (cluster, namespace, workload) (
        count_over_time(
          namespace_workload_pod:kube_pod_owner:relabel{workload_type="deployment", namespace=~"ocm-.*"}[1m]
        )
      )

Files reviewed: 1/1 changed files
Comments generated: 2
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

    unit: percent
    workspace: svc
    step: "60s"
+  - title: "HCP Control Plane Pod Churn Ratio"


    workspace: svc
    step: "60s"
+  - title: "HCP Control Plane Pod Churn Ratio"
+    description: "Distinct pods created per Deployment, divided by desired replica count. ~1 = healthy; higher means pods were repeatedly replaced, not just scaled up. Covers HostedControlPlane pods, which normally churn heavily in their first few minutes of life."


Copilot

🟡 Changes recommended

The new PromQL query should reuse the existing pod→workload recording rule and guard against divide-by-zero to avoid incorrect/noisy results.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Files reviewed: 1/1 changed files
Comments generated: 2
Review effort level: Lite

+      count by (cluster, namespace, workload) (
+        count_over_time(
+          label_replace(
+            label_replace(
+              max without (prometheus_replica) (
+                kube_pod_owner{job="kube-state-metrics", owner_kind="ReplicaSet", namespace=~"ocm-.*"}
+              ),
+              "replicaset", "$1", "owner_name", "(.*)"
+            )
+            * on (replicaset, namespace) group_left(owner_name) topk by (replicaset, namespace) (
+              1, max without (prometheus_replica) (
+                kube_replicaset_owner{job="kube-state-metrics", owner_kind="Deployment", namespace=~"ocm-.*"}
+              )
+            ),
+            "workload", "$1", "owner_name", "(.*)"
+          )[1m:30s]
+        )
+      )


+      / on (cluster, namespace, workload)
+      max by (cluster, namespace, workload) (
+        label_replace(
+          max_over_time(
+            kube_deployment_spec_replicas{job="kube-state-metrics", namespace=~"ocm-.*"}[1m]
+          ),
+          "workload", "$1", "deployment", "(.*)"
+        )
+      )


Copilot

🔵 Needs a closer look

The PR description is missing required PR-standards items (tracking ticket link and screenshots for a metrics visualization change), and there are also query/description inconsistencies captured in review comments.

Review details

Suppressed comments (2)

test/cmd/aro-hcp-tests/gather-observability/queries.yaml:447

The description says this is scoped to HostedControlPlane Deployments, but the PromQL filter is namespace=~"ocm-.*", which also matches the HostedCluster namespace pattern (ocm-<prefix>-<cluster-id>). Either tighten the query to control-plane namespaces or adjust the description to match what the query actually selects.

    description: "Highest pod churn ratio seen anywhere on each management cluster: distinct pods observed per Deployment, divided by desired replica count, maxed across every HostedControlPlane Deployment on that cluster. ~1 = healthy; higher means at least one Deployment somewhere on the cluster had pods repeatedly replaced, not just scaled up. Copy the query below and drop the outer max-by-cluster to drill into which HCP instance and component is churning."

test/cmd/aro-hcp-tests/gather-observability/queries.yaml:455

The PR description says this query reuses the existing namespace_workload_pod:kube_pod_owner:relabel recording rule, but the current PromQL re-derives the pod→ReplicaSet→Deployment join inline. Using the recording rule would better match the stated intent and reduce query complexity/cost.

          count_over_time(
            label_replace(
              label_replace(
                max without (prometheus_replica) (
                  kube_pod_owner{job="kube-state-metrics", owner_kind="ReplicaSet", namespace=~"ocm-.*"}

Files reviewed: 1/1 changed files
Comments generated: 0 new
Review effort level: Lite

Copilot

🔵 Needs a closer look

The new query currently turns the replica-count denominator into a boolean (via > 0), which breaks the intended churn ratio calculation.

Review details

Suppressed comments (3)

test/cmd/aro-hcp-tests/gather-observability/queries.yaml:454

The PR description says this query reuses the existing namespace_workload_pod:kube_pod_owner:relabel recording rule, but the query currently re-derives the ReplicaSet→Deployment join inline. Using the recording rule here would be simpler and should be significantly cheaper at query time (especially when running gather-observability over a large time range).

        count by (cluster, namespace, workload) (
          count_over_time(
            label_replace(
              label_replace(
                max without (prometheus_replica) (

test/cmd/aro-hcp-tests/gather-observability/queries.yaml:477

The denominator currently ends with ) > 0, which converts the replica count vector into a boolean (0/1) and breaks the churn ratio (it effectively stops dividing by desired replicas). If the goal is to avoid divide-by-zero, clamp the replica count to a minimum of 1 (or filter separately) while preserving the actual replica values.

          max by (cluster, namespace, workload) (
            label_replace(
              max_over_time(
                kube_deployment_spec_replicas{job="kube-state-metrics", namespace=~"ocm-.*"}[1m]
              ),
              "workload", "$1", "deployment", "(.*)"
            )
          ) > 0

test/cmd/aro-hcp-tests/gather-observability/queries.yaml:447

This change adds a new user-visible metrics visualization in gather-observability. CONTRIBUTING.md requires before/after screenshots for dashboard/graph/metrics visualization changes; please add a screenshot of the rendered panel (e.g., from the Spyglass artifact) to the PR description before merge.

  - title: "HCP Control Plane Pod Churn Ratio"
    description: "Highest pod churn ratio seen anywhere on each management cluster: distinct pods observed per Deployment, divided by desired replica count, maxed across every HostedControlPlane Deployment on that cluster. ~1 = healthy; higher means at least one Deployment somewhere on the cluster had pods repeatedly replaced, not just scaled up. Copy the query below and drop the outer max-by-cluster to drill into which HCP instance and component is churning."

Files reviewed: 1/1 changed files
Comments generated: 0 new
Review effort level: Lite

Gerd Oberlechner (geoberle) · 2026-09-05T14:15:20Z

/retest

Adds an "MC AKS Metrics" panel with a query reporting distinct pods created per Deployment vs desired replica count for HostedControlPlane namespaces, to surface heavy pod replacement (e.g. from node instability) during a HCP's short lifetime.

namespace_workload_pod:kube_pod_owner:relabel is only wired up as an Azure Monitor recording rule for the svc workspace, not hcp, so the panel rendered empty. Do the pod->ReplicaSet->Deployment join directly against raw kube-state-metrics series instead, which do reach the hcp workspace. Also fixes the panel description, which claimed to count "pods created" when it actually counts distinct pods observed in the window.

A real e2e-parallel run has ~55 concurrent HCP namespaces x ~44 control-plane Deployments x 3 mgmt clusters, producing ~1635 series on one un-faceted line chart -- unreadable. Wrap the existing per-(cluster,namespace,workload) ratio in max by (cluster) so the chart shows one line per management cluster: the worst churn ratio seen anywhere on it. Drop the outer max to drill into a specific HCP instance/component.

kube_deployment_spec_replicas can be 0 (or absent) for a HostedControlPlane Deployment during teardown, while its pods are still being observed -- dividing by that produced +Inf, serialized as math.MaxFloat64, which poisoned the per-cluster max and rendered the whole chart as a flat line pinned near zero on a 10^308-wide axis. Filter the denominator to > 0 so those samples are dropped from the division instead of producing infinity. Confirmed live via Playwright against the actual CI artifact -- an earlier attempt at "fixing" the chart renderer template was based on inspecting the wrong chart (several queries in this panel share identical cluster-only legend labels) and has been reverted.

Copilot

🔵 Needs a closer look

The new query should be simplified to use the existing pod→workload recording rule (to reduce cost/complexity) and its description should be corrected to match the query’s actual namespace scope.

Review details

Suppressed comments (2)

test/cmd/aro-hcp-tests/gather-observability/queries.yaml:498

The description says this is maxed across every "HostedControlPlane Deployment" / "HostedControlPlane namespaces", but the query filters only on namespace=~"ocm-.*", which includes both HostedCluster and control-plane namespaces. Either tighten the namespace selection to control-plane namespaces only, or update the description so it matches the actual scope of the query.

    description: "Highest pod churn ratio seen anywhere on each management cluster: distinct pods observed per Deployment, divided by desired replica count, maxed across every HostedControlPlane Deployment on that cluster. ~1 = healthy; higher means at least one Deployment somewhere on the cluster had pods repeatedly replaced, not just scaled up. Copy the query below and drop the outer max-by-cluster to drill into which HCP instance and component is churning."

test/cmd/aro-hcp-tests/gather-observability/queries.yaml:506

This query re-derives the ReplicaSet→Deployment join inline, which is expensive (subquery + joins) and duplicates the existing namespace_workload_pod:kube_pod_owner:relabel recording rule already deployed for exactly this pod→Deployment mapping. Using the recording rule here avoids the subquery ([1m:30s]) and reduces query cost/cardinality while keeping the same semantics.

          count_over_time(
            label_replace(
              label_replace(
                max without (prometheus_replica) (
                  kube_pod_owner{job="kube-state-metrics", owner_kind="ReplicaSet", namespace=~"ocm-.*"}

Files reviewed: 1/1 changed files
Comments generated: 0 new
Review effort level: Lite

Grouping by cluster alone hid which HCP instance was actually churning (a max collapses everything into 3 flat lines). Regroup by (cluster, namespace), filter to genuine churn (ratio > 1 -- our ratio is an exact rational, so this cleanly excludes the healthy baseline), and take topk(10) of what's left so the chart stays readable regardless of how many HCP namespaces are live. Top-10 membership is recomputed per step, so it can shift minute to minute.

Copilot

🔵 Needs a closer look

The new query does not actually use the stated namespace_workload_pod:kube_pod_owner:relabel recording rule and the panel description does not match the query’s namespace selector.

Review details

Suppressed comments (2)

test/cmd/aro-hcp-tests/gather-observability/queries.yaml:507

The PR description says this query “reuses the existing namespace_workload_pod:kube_pod_owner:relabel recording rule”, but the current implementation re-derives the pod→ReplicaSet→Deployment join inline via kube_pod_owner/kube_replicaset_owner and label_replace. Using the recording rule here would both match the PR summary and materially simplify the query (less duplication and lower query cost).

            count by (cluster, namespace, workload) (
              count_over_time(
                label_replace(
                  label_replace(
                    max without (prometheus_replica) (

test/cmd/aro-hcp-tests/gather-observability/queries.yaml:498

The description says “HostedControlPlane namespaces only”, but the query selectors use namespace=~"ocm-.", which also matches other ocm- namespaces. Either tighten the selector to match only HostedControlPlane namespaces, or adjust the description so it matches what the query actually does.

    description: "Top 10 HostedControlPlane namespaces by worst pod churn ratio this minute: distinct pods observed per Deployment, divided by desired replica count, maxed across every Deployment in that namespace. Only namespaces above the healthy baseline of 1 are ranked, so a quiet minute can show fewer than 10 lines (or none). Top-10 membership is recomputed every minute, so which namespaces appear can shift over time. Copy the query below and drop the outer topk/max-by-namespace to drill into which component is churning."

Files reviewed: 1/1 changed files
Comments generated: 0 new
Review effort level: Lite

Per-minute topk let ties (exact-rational ratios) and per-step reshuffling push the legend past 10 and made membership flap between adjacent minutes. Rank instead by each namespace's peak ratio over the whole 6h lookback (and on (cluster, namespace) against the live per-minute value), so membership only changes when a genuinely bigger spike appears, while still plotting the real per-minute value. This duplicates the pod/Deployment join (PromQL has no way to reuse a sub-result) and evaluates it at every sub-point of the 6h ranking window, making this by far the most expensive query in the panel -- noted in the description as the first thing to check if gather-observability CI runs slow down or start timing out.

Copilot

🟡 Changes recommended

The new query contradicts the PR description by re-deriving (and duplicating) the pod→Deployment join instead of using the stated recording rule, and the PR is also missing required PR-standards items (tracking ticket link and screenshots for the new visualization).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Files reviewed: 1/1 changed files
Comments generated: 1
Review effort level: Lite

+              label_replace(
+                label_replace(
+                  max without (prometheus_replica) (
+                    kube_pod_owner{job="kube-state-metrics", owner_kind="ReplicaSet", namespace=~"ocm-.*"}
+                  ),


openshift-ci · 2026-09-06T17:28:39Z

Gerd Oberlechner (@geoberle): The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-parallel	`48178a8`	link	true	`/test e2e-parallel`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Copilot AI lite review requested due to automatic review settings September 4, 2026 16:36

openshift-ci Bot added the approved label Sep 4, 2026

openshift-ci Bot requested review from Mike Gahagan (mgahagan73) and Patrik Suba (patriksuba) September 4, 2026 16:36

Copilot started reviewing on behalf of Gerd Oberlechner (geoberle) September 4, 2026 16:36 View session

Copilot AI reviewed Sep 4, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings September 5, 2026 05:41

Copilot started reviewing on behalf of Gerd Oberlechner (geoberle) September 5, 2026 05:41 View session

Copilot AI reviewed Sep 5, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings September 5, 2026 08:29

Copilot started reviewing on behalf of Gerd Oberlechner (geoberle) September 5, 2026 08:29 View session

Copilot AI reviewed Sep 5, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings September 5, 2026 14:04

Copilot started reviewing on behalf of Gerd Oberlechner (geoberle) September 5, 2026 14:04 View session

Copilot AI reviewed Sep 5, 2026

View reviewed changes

Gerd Oberlechner (geoberle) added 4 commits September 5, 2026 17:05

Copilot AI review requested due to automatic review settings September 5, 2026 16:45

Gerd Oberlechner (geoberle) force-pushed the pod-churn branch from 522ebe6 to 0e0b562 Compare September 5, 2026 16:45

Copilot started reviewing on behalf of Gerd Oberlechner (geoberle) September 5, 2026 16:46 View session

Copilot AI reviewed Sep 5, 2026

View reviewed changes

Rael Garcia (raelga) mentioned this pull request Sep 6, 2026

fix(config): raise ci00/ci01 CosmosDB resourceContainerMaxScale to 20000 (AROSLSRE-2037) #6844

Closed

Copilot AI review requested due to automatic review settings September 6, 2026 11:29

Copilot started reviewing on behalf of Gerd Oberlechner (geoberle) September 6, 2026 11:30 View session

Copilot AI reviewed Sep 6, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings September 6, 2026 15:15

Copilot started reviewing on behalf of Gerd Oberlechner (geoberle) September 6, 2026 15:16 View session

Copilot AI reviewed Sep 6, 2026

View reviewed changes

Comment thread test/cmd/aro-hcp-tests/gather-observability/queries.yaml

Comment on lines +504 to +508

label_replace(

label_replace(

max without (prometheus_replica) (

kube_pod_owner{job="kube-state-metrics", owner_kind="ReplicaSet", namespace=~"ocm-.*"}

),

Navigation Menu

Uh oh!

feat: add HCP pod churn ratio panel to gather-observability - #6836

feat: add HCP pod churn ratio panel to gather-observability#6836
Gerd Oberlechner (geoberle) wants to merge 6 commits into
Azure:mainfrom
geoberle:pod-churn

Gerd Oberlechner (geoberle) commented Sep 4, 2026

Uh oh!

openshift-ci Bot commented Sep 4, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Gerd Oberlechner (geoberle) commented Sep 5, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

openshift-ci Bot commented Sep 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Gerd Oberlechner (geoberle) commented Sep 4, 2026

Summary

Test plan

Uh oh!

openshift-ci Bot commented Sep 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (1)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Suppressed comments (2)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Suppressed comments (3)

Uh oh!

Gerd Oberlechner (geoberle) commented Sep 5, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Suppressed comments (2)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Suppressed comments (2)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

openshift-ci Bot commented Sep 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants