feat(observability): add SLI SLO metrics dashboard

[Simon Wiencki (swiencki)]

What

Adds an ARO HCP SLI / SLO Metrics dashboard to the SRE User Journey folder.

The dashboard provides a persistent Grafana view of the main Prometheus-backed sections and charts from the MCP Chat sli-slo-metrics report:

• Cluster create success rate, p95 duration, failures, and ARM breaches
• Create outcomes and duration percentiles for clusters, NodePools, and ExternalAuth resources
• Rolling seven-day install volume, success-rate, and install-time trends
• Create, delete, and update outcomes
• Current phases, stuck-operation severity, and ARM breach details
• Install-version and control-plane version distributions

Tracking

Jira: ARO-29463

This dashboard is the follow-up to the SLI/SLO install-volume investigation and moves the existing report views into Grafana for ongoing use.

Data model

The backend operation metrics are current-operation gauges. Live panels select terminal operations whose latest transition falls inside the chosen lookback. The rolling seven-day charts show the last six hours of those rolling values so the queries remain within Azure Managed Prometheus limits.

The generated report remains authoritative for start-time-bounded, deduplicated historical cohorts. The dashboard also calls out two sections that Prometheus cannot reproduce:

• Normalized failure reasons require Kusto enrichment.
• Customer incident totals require IcM or Kusto.

Screenshots

This is a new dashboard, so there is no before state. The screenshots below are a local layout preview with illustrative mock values. The production Prometheus queries were tested separately against the services-uksouth and hcps-uksouth datasources.

Validation

• make -C observability verify
• Production PromQL probes for summary, create outcomes, percentiles, rolling trends, operation outcomes, stuck operations, install versions, and control-plane versions

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: swiencki

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• observability/OWNERS [swiencki]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Copilot]

🟡 Changes recommended

Several dashboard queries/settings are likely to render empty or misleading panels (notably the default exclude_subs behavior and overly coarse interval: "7d" on 6h trend panels).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a new SRE “User Journey” Grafana dashboard (ARO HCP SLI / SLO Metrics) to provide a persistent, Prometheus-backed view of the key sections from the sli-slo-metrics report (create health, rolling trends, stuck ops, and version distributions).

Changes:

• Introduces a new Grafana dashboard JSON with panels for create success rate/durations, outcomes, and rolling 7-day trends.
• Adds stuck-operation inventory/severity panels and an ARM breach details table.
• Adds datasource and filtering variables (services datasource selector, HCP datasource selector, subscription-exclude textbox).

File summaries

File	Description
observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json	New dashboard definition for live SLI/SLO metrics and related operational views

Review details

Suppressed comments (5)

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:1468

• This stuck-operation series doesn't apply the dashboard's exclude_subs filter and isn't scoped to the dashboard's resource/operation types, so the severity totals can include unrelated operations and become noisy.

          "expr": "count((backend_resource_operation_phase_info{phase!~\"succeeded|failed|canceled\"} == 1) and ((time() - backend_resource_operation_last_transition_time_seconds{phase!~\"succeeded|failed|canceled\"}) > 3600)) or vector(0)",

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:881

• The NodePool rolling 7-day volume panel also sets the Prometheus query interval to 7d, which will collapse the 6h panel view to ~one point. Use a smaller interval (e.g. 5m) so the trend renders as intended.

          "expr": "count((backend_resource_operation_phase_info{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters/nodepools\",operation_type=\"create\",phase=\"succeeded\",subscription_id!~\"${exclude_subs:raw}\"} == 1) and ((time() - backend_resource_operation_last_transition_time_seconds{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters/nodepools\",operation_type=\"create\",phase=\"succeeded\",subscription_id!~\"${exclude_subs:raw}\"}) < 604800))",
          "instant": false,
          "interval": "7d",
          "legendFormat": "Succeeded",
          "range": true,

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:996

• This success-rate trend panel sets the query interval to 7d, which will collapse the 6h timeFrom view to ~one sample. Use a smaller step (e.g. 5m) so the trend line is meaningful.

          "expr": "100 * count((backend_resource_operation_phase_info{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters\",operation_type=\"create\",phase=\"succeeded\",subscription_id!~\"${exclude_subs:raw}\"} == 1) and ((time() - backend_resource_operation_last_transition_time_seconds{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters\",operation_type=\"create\",phase=\"succeeded\",subscription_id!~\"${exclude_subs:raw}\"}) < 604800)) / clamp_min(count((backend_resource_operation_phase_info{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters\",operation_type=\"create\",phase=~\"succeeded|failed\",subscription_id!~\"${exclude_subs:raw}\"} == 1) and ((time() - backend_resource_operation_last_transition_time_seconds{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters\",operation_type=\"create\",phase=~\"succeeded|failed\",subscription_id!~\"${exclude_subs:raw}\"}) < 604800)), 1)",
          "instant": false,
          "interval": "7d",
          "legendFormat": "Cluster",
          "range": true,

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:1109

• This panel’s Prometheus queries use interval: "7d", which is too coarse for a 6h timeFrom view and will usually collapse the percentile trend. Use a smaller interval (e.g. 5m) so the line chart can show change over time.

          "expr": "quantile(0.50, ((backend_resource_operation_last_transition_time_seconds{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters\",operation_type=\"create\",phase=\"succeeded\",subscription_id!~\"${exclude_subs:raw}\"} - backend_resource_operation_start_time_seconds{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters\",operation_type=\"create\",phase=\"succeeded\",subscription_id!~\"${exclude_subs:raw}\"}) / 60) and ((time() - backend_resource_operation_last_transition_time_seconds{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters\",operation_type=\"create\",phase=\"succeeded\",subscription_id!~\"${exclude_subs:raw}\"}) < 604800))",
          "instant": false,
          "interval": "7d",
          "legendFormat": "Cluster p50",
          "range": true,

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:1135

• The NodePool percentile series in this panel also uses interval: "7d", which is too coarse for the 6h view and will usually collapse the trend. Use a smaller interval (e.g. 5m).

          "expr": "quantile(0.50, ((backend_resource_operation_last_transition_time_seconds{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters/nodepools\",operation_type=\"create\",phase=\"succeeded\",subscription_id!~\"${exclude_subs:raw}\"} - backend_resource_operation_start_time_seconds{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters/nodepools\",operation_type=\"create\",phase=\"succeeded\",subscription_id!~\"${exclude_subs:raw}\"}) / 60) and ((time() - backend_resource_operation_last_transition_time_seconds{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters/nodepools\",operation_type=\"create\",phase=\"succeeded\",subscription_id!~\"${exclude_subs:raw}\"}) < 604800))",
          "instant": false,
          "interval": "7d",
          "legendFormat": "NodePool p50",
          "range": true,

• Files reviewed: 1/1 changed files
• Comments generated: 5
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Simon Wiencki (swiencki)]

Addressed all Copilot findings in 430685d4f:

• default exclusion regex now includes all subscriptions
• operation inventory and stuck panels are scoped to HCP resource and operation types and honor exclusions
• rolling trend intervals are now 5 minutes; the production probe returned 73 points over 6 hours

observability-prometheus and test-unit failed only because of pod scheduling timeouts.

/retest

Copilot review

[Copilot]

🟡 Changes recommended

The “Cluster Create Success Rate” stat can produce NaN/Inf due to a zero denominator (and the PR description also appears to be missing a required tracking ticket/issue link per CONTRIBUTING.md).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 1/1 changed files
• Comments generated: 2
• Review effort level: Lite

[Simon Wiencki (swiencki)]

Addressed the second Copilot review in 0128fb3ce: the terminal denominator is clamped and the datasource guidance now matches the dropdown names. The PR body also explains why there is no separate tracking ticket.

The remaining e2e-parallel failure is unrelated to this dashboard-only change. The run hit backend 404s for subscription deployment and operation-status resources during cluster tests. Retrying it now.

/retest

Copilot review

[Copilot]

🟡 Changes recommended

Several dashboard queries compute success-rate denominators without canceled and label “ARM breach / severity” using last-transition time instead of operation start time, which can materially misreport SLO/SLA health.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (8)

Previously missed (3) — in code that hasn't changed since the last review.

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:395

• This panel labels the result as an "ARM breach" but measures time since last transition rather than time since operation start. That can miss operations that have progressed recently but have still exceeded the 60-minute ARM completion threshold. Use backend_resource_operation_start_time_seconds (consistent with the existing SLO dashboards) to measure total operation duration.
  observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:1457
• This severity bucket uses time since last transition; for the stated "SLO breach >20m" threshold, it should reflect time since operation start. Using last-transition time can undercount long-running operations that continue to make progress. Consider switching to time() - backend_resource_operation_start_time_seconds for these buckets.

This issue also appears in the following locations of the same file:

• line 1468
• line 1479
• line 1490
  observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:1605

• The table is described as an ARM breach detail view, but the value is computed from last transition time, not time since operation start. That can produce misleading "Duration (min)" values for long-running operations that have progressed recently. Use backend_resource_operation_start_time_seconds for the duration calculation and filter.

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:992

• This weekly cluster success-rate denominator excludes phase="canceled", which can overstate the success rate when canceled operations exist. Treat canceled as terminal in the denominator (consistent with other SLO dashboards in this repo).

          "expr": "100 * count((backend_resource_operation_phase_info{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters\",operation_type=\"create\",phase=\"succeeded\",subscription_id!~\"${exclude_subs:raw}\"} == 1) and ((time() - backend_resource_operation_last_transition_time_seconds{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters\",operation_type=\"create\",phase=\"succeeded\",subscription_id!~\"${exclude_subs:raw}\"}) < 604800)) / clamp_min(count((backend_resource_operation_phase_info{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters\",operation_type=\"create\",phase=~\"succeeded|failed\",subscription_id!~\"${exclude_subs:raw}\"} == 1) and ((time() - backend_resource_operation_last_transition_time_seconds{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters\",operation_type=\"create\",phase=~\"succeeded|failed\",subscription_id!~\"${exclude_subs:raw}\"}) < 604800)), 1)",

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:1005

• This weekly NodePool success-rate denominator excludes phase="canceled", which can overstate the success rate when canceled operations exist. Treat canceled as terminal in the denominator (consistent with other SLO dashboards in this repo).

          "expr": "100 * count((backend_resource_operation_phase_info{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters/nodepools\",operation_type=\"create\",phase=\"succeeded\",subscription_id!~\"${exclude_subs:raw}\"} == 1) and ((time() - backend_resource_operation_last_transition_time_seconds{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters/nodepools\",operation_type=\"create\",phase=\"succeeded\",subscription_id!~\"${exclude_subs:raw}\"}) < 604800)) / clamp_min(count((backend_resource_operation_phase_info{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters/nodepools\",operation_type=\"create\",phase=~\"succeeded|failed\",subscription_id!~\"${exclude_subs:raw}\"} == 1) and ((time() - backend_resource_operation_last_transition_time_seconds{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters/nodepools\",operation_type=\"create\",phase=~\"succeeded|failed\",subscription_id!~\"${exclude_subs:raw}\"}) < 604800)), 1)",

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:1490

• This severity bucket uses time since last transition; "Zombie >7d" is more accurate when measured as time since operation start. Using last-transition time can undercount operations that have been running for >7d but have continued to transition phases.

          "expr": "count((backend_resource_operation_phase_info{resource_type=~\"microsoft.redhatopenshift/hcpopenshiftclusters($|/nodepools$|/externalauths$)\",operation_type=~\"create|delete|update\",phase!~\"succeeded|failed|canceled\",subscription_id!~\"${exclude_subs:raw}\"} == 1) and ((time() - backend_resource_operation_last_transition_time_seconds{resource_type=~\"microsoft.redhatopenshift/hcpopenshiftclusters($|/nodepools$|/externalauths$)\",operation_type=~\"create|delete|update\",phase!~\"succeeded|failed|canceled\",subscription_id!~\"${exclude_subs:raw}\"}) > 604800)) or vector(0)",

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:1468

• This severity bucket uses time since last transition; for the stated "ARM breach >60m" threshold, it should reflect time since operation start. Using last-transition time can miss long-running operations that have advanced phases recently.

          "expr": "count((backend_resource_operation_phase_info{resource_type=~\"microsoft.redhatopenshift/hcpopenshiftclusters($|/nodepools$|/externalauths$)\",operation_type=~\"create|delete|update\",phase!~\"succeeded|failed|canceled\",subscription_id!~\"${exclude_subs:raw}\"} == 1) and ((time() - backend_resource_operation_last_transition_time_seconds{resource_type=~\"microsoft.redhatopenshift/hcpopenshiftclusters($|/nodepools$|/externalauths$)\",operation_type=~\"create|delete|update\",phase!~\"succeeded|failed|canceled\",subscription_id!~\"${exclude_subs:raw}\"}) > 3600)) or vector(0)",

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:1479

• This severity bucket uses time since last transition; "Severe >24h" is more accurate when measured as time since operation start. Using last-transition time can undercount operations that have been running for >24h but progressed recently.

          "expr": "count((backend_resource_operation_phase_info{resource_type=~\"microsoft.redhatopenshift/hcpopenshiftclusters($|/nodepools$|/externalauths$)\",operation_type=~\"create|delete|update\",phase!~\"succeeded|failed|canceled\",subscription_id!~\"${exclude_subs:raw}\"} == 1) and ((time() - backend_resource_operation_last_transition_time_seconds{resource_type=~\"microsoft.redhatopenshift/hcpopenshiftclusters($|/nodepools$|/externalauths$)\",operation_type=~\"create|delete|update\",phase!~\"succeeded|failed|canceled\",subscription_id!~\"${exclude_subs:raw}\"}) > 86400)) or vector(0)",

• Files reviewed: 1/1 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

Several PromQL queries appear to mis-handle terminal phases and duration semantics (and the datasource regex is overly restrictive), which can materially skew the dashboard’s reported SLO/ARM-breach signals.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (4)

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:116

• This terminal-count denominator excludes canceled, which is treated as a terminal phase in existing SLO recording rules (e.g. observability/recording-rules/cluster-provision-slo-recordingRule.yaml:45-49). Excluding it can inflate the displayed success rate; include canceled in the terminal phase set.

          "expr": "clamp_min(count((backend_resource_operation_phase_info{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters\",operation_type=\"create\",phase=~\"succeeded|failed\",subscription_id!~\"${exclude_subs:raw}\"} == 1) and ((time() - backend_resource_operation_last_transition_time_seconds{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters\",operation_type=\"create\",phase=~\"succeeded|failed\",subscription_id!~\"${exclude_subs:raw}\"}) < $__range_s)), 1)",

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:1079

• This "Cluster Terminal" query excludes canceled, but canceled is considered terminal elsewhere in the repo’s SLO logic. Include canceled so the success-rate calculation matches the SLO definition.

          "expr": "count((backend_resource_operation_phase_info{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters\",operation_type=\"create\",phase=~\"succeeded|failed\",subscription_id!~\"${exclude_subs:raw}\"} == 1) and ((time() - backend_resource_operation_last_transition_time_seconds{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters\",operation_type=\"create\",phase=~\"succeeded|failed\",subscription_id!~\"${exclude_subs:raw}\"}) < $__range_s))",

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:1105

• This "NodePool Terminal" query excludes canceled, but canceled is treated as terminal in the repo’s SLO logic. Include canceled so the NodePool success-rate line isn’t inflated.

          "expr": "count((backend_resource_operation_phase_info{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters/nodepools\",operation_type=\"create\",phase=~\"succeeded|failed\",subscription_id!~\"${exclude_subs:raw}\"} == 1) and ((time() - backend_resource_operation_last_transition_time_seconds{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters/nodepools\",operation_type=\"create\",phase=~\"succeeded|failed\",subscription_id!~\"${exclude_subs:raw}\"}) < $__range_s))",

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:1817

• These "ARM breach" panels compute duration from backend_resource_operation_last_transition_time_seconds, which measures time since the last phase change, not total operation duration. The repo’s alerting rules use backend_resource_operation_start_time_seconds for ARM-threshold breaches (e.g. observability/alerts/access-cluster-slo-prometheusRule.yaml:113-123). To align the dashboard with the ARM threshold semantics, base the duration on *_start_time_seconds.

          "id": "merge",

• Files reviewed: 1/1 changed files
• Comments generated: 2
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

The dashboard datasource regexes can exclude valid regional datasources, and some success-rate calculations can produce inflated/NaN results due to terminal-phase semantics and divide-by-zero.

Review details

Suppressed comments (5)

Previously missed (1) — in code that hasn't changed since the last review.

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:1105

• Same as the cluster series: the NodePool terminal count can be 0 at some steps, producing NaN/Inf in the calculated success-rate field. Clamp the terminal count and include canceled as a terminal phase.

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:116

• The success-rate denominator excludes phase="canceled", which can inflate the reported success rate if creates are canceled (this dashboard already treats canceled as an outcome in other panels). Consider counting succeeded|failed|canceled as terminal here so canceled creates reduce the success rate.

          "expr": "clamp_min(count((backend_resource_operation_phase_info{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters\",operation_type=\"create\",phase=~\"succeeded|failed\",subscription_id!~\"${exclude_subs:raw}\"} == 1) and ((time() - backend_resource_operation_last_transition_time_seconds{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters\",operation_type=\"create\",phase=~\"succeeded|failed\",subscription_id!~\"${exclude_subs:raw}\"}) < $__range_s)), 1)",

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:1079

• This time-series panel divides by the "Cluster Terminal" series, but the terminal count can be 0 for some steps, which will yield NaN/Inf in the calculated field. Clamping the terminal count to at least 1 (and including canceled as terminal) avoids divide-by-zero artifacts and aligns terminal semantics with other panels.

          "expr": "100 * count((backend_resource_operation_phase_info{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters/nodepools\",operation_type=\"create\",phase=\"succeeded\",subscription_id!~\"${exclude_subs:raw}\"} == 1) and ((time() - backend_resource_operation_last_transition_time_seconds{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters/nodepools\",operation_type=\"create\",phase=\"succeeded\",subscription_id!~\"${exclude_subs:raw}\"}) < $__range_s)) / clamp_min(count((backend_resource_operation_phase_info{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters/nodepools\",operation_type=\"create\",phase=~\"succeeded|failed\",subscription_id!~\"${exclude_subs:raw}\"} == 1) and ((time() - backend_resource_operation_last_transition_time_seconds{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters/nodepools\",operation_type=\"create\",phase=~\"succeeded|failed\",subscription_id!~\"${exclude_subs:raw}\"}) < $__range_s)), 1)",

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:2134

• The datasource selector regex is overly restrictive (letters-only region names). This will exclude valid Azure regions/datasources that contain digits (e.g. ...-eastus2). Other dashboards in this repo use ^Managed_Prometheus_services-.*$ to avoid filtering out regions.
  observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:2147
• Same issue as the services datasource selector: this regex can exclude valid HCP datasources for regions with digits in the name. Align to the repo pattern ^Managed_Prometheus_hcps-.*$ so all regions remain selectable.

• Files reviewed: 1/1 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

Several PromQL expressions and datasource variable regexes will skew or drop data (especially when using “All” datasources), risking incorrect dashboard results.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (5)

Previously missed (1) — in code that hasn't changed since the last review.

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:38

• The PR description claims this dashboard includes rolling seven-day install-volume/success-rate/install-time trends and ARM-breach details, but this JSON doesn’t appear to include any panels/strings for rolling 7d trends or ARM breach views. Please either add the missing panels or update the PR description to match what’s actually being shipped in this dashboard.

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:1066

• This success-rate timeseries computes a mean across regional datasources in Grafana. Using clamp_min(..., 1) forces regions with zero terminal ops to appear as 0% and drags the mean down. Instead, drop the series entirely when terminal count is 0 so the mean only includes regions with activity.

          "expr": "100 * count((backend_resource_operation_phase_info{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters\",operation_type=\"create\",phase=\"succeeded\",subscription_id!~\"${exclude_subs:raw}\"} == 1) and ((time() - backend_resource_operation_last_transition_time_seconds{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters\",operation_type=\"create\",phase=\"succeeded\",subscription_id!~\"${exclude_subs:raw}\"}) < $__range_s)) / clamp_min(count((backend_resource_operation_phase_info{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters\",operation_type=\"create\",phase=~\"succeeded|failed\",subscription_id!~\"${exclude_subs:raw}\"} == 1) and ((time() - backend_resource_operation_last_transition_time_seconds{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters\",operation_type=\"create\",phase=~\"succeeded|failed\",subscription_id!~\"${exclude_subs:raw}\"}) < $__range_s)), 1)",

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:1079

• Same issue as the Cluster series above: clamp_min(..., 1) makes regions with zero NodePool creates show as 0% and skews the cross-datasource mean. Drop the series when the terminal count is 0.

          "expr": "100 * count((backend_resource_operation_phase_info{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters/nodepools\",operation_type=\"create\",phase=\"succeeded\",subscription_id!~\"${exclude_subs:raw}\"} == 1) and ((time() - backend_resource_operation_last_transition_time_seconds{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters/nodepools\",operation_type=\"create\",phase=\"succeeded\",subscription_id!~\"${exclude_subs:raw}\"}) < $__range_s)) / clamp_min(count((backend_resource_operation_phase_info{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters/nodepools\",operation_type=\"create\",phase=~\"succeeded|failed\",subscription_id!~\"${exclude_subs:raw}\"} == 1) and ((time() - backend_resource_operation_last_transition_time_seconds{resource_type=\"microsoft.redhatopenshift/hcpopenshiftclusters/nodepools\",operation_type=\"create\",phase=~\"succeeded|failed\",subscription_id!~\"${exclude_subs:raw}\"}) < $__range_s)), 1)",

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:2070

• The datasource variable regex is more restrictive than other dashboards in this repo and will exclude valid AMW datasource names for regions that include digits (e.g. eastus2). Other dashboards use ^Managed_Prometheus_services-.*$ (e.g. observability/grafana-dashboards/sre/user-journey/mgmt-cluster-triage.json:1665).

        "regex": "^Managed_Prometheus_services-[a-z]{4,}.*$",

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:2083

• Same as the service-cluster datasource variable: this regex is more restrictive than the established pattern and will exclude hcps datasources for regions with digits. Other dashboards use ^Managed_Prometheus_hcps-.*$ (e.g. observability/grafana-dashboards/kas-monitor/kas_golden_signals.json:1878).

        "regex": "^Managed_Prometheus_hcps-[a-z]{4,}.*$",

• Files reviewed: 1/1 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟢 Approval recommended

The change is a self-contained new dashboard addition with no functional code-path impact; only a small usability/doc clarification was noted.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:2094

• The exclude_subs variable description doesn’t mention that subscription_id metric labels are lowercased in the backend (see subscriptionIDMetricLabel), so pasting an uppercase GUID won’t match and exclusions may silently fail. Clarify that the input should be lowercase (or explicitly a regex) to avoid confusing dashboard users.

• Files reviewed: 1/1 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Simon Wiencki (swiencki)]

/retest ci/prow/e2e-parallel

[Simon Wiencki (swiencki)]

/test e2e-parallel

[Simon Wiencki (swiencki)]

/retest

[Copilot]

🟢 Approval recommended

The change is a new dashboard addition with no security-sensitive modifications, and only minor follow-up nits were identified.

Review details

Suppressed comments (1)

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:2384

• Datasource selector regex uses .+$ which is inconsistent with other dashboards in this folder (they use .*$). Aligning avoids accidental exclusion if a datasource ever has an empty suffix.

        "query": "prometheus",
        "refresh": 1,
        "regex": "^Managed_Prometheus_hcps-.+$",
        "type": "datasource"

• Files reviewed: 1/1 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟢 Approval recommended

The change is a self-contained new dashboard with no functional code-path impact, and the only feedback is minor consistency nits in datasource regex patterns.

Review details

Suppressed comments (2)

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:2372

• The HCP datasource variable regex uses .+$, while other dashboards in this repo consistently use .*$. Aligning avoids unnecessary divergence and is slightly more permissive.

        "regex": "^Managed_Prometheus_hcps-.+$",

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:2359

• The datasource variable regex uses .+$, while other dashboards in this repo consistently use .*$ (e.g. cluster-provisioning-slo.json). Aligning avoids unnecessary divergence and is slightly more permissive (won’t exclude an empty suffix).

        "regex": "^Managed_Prometheus_services-.+$",

• Files reviewed: 1/1 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

The new dashboard’s datasource selector regexes should align with the repo-wide convention (.*$) to avoid unintentionally excluding valid Managed Prometheus datasources.

Review details

Suppressed comments (2)

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:2229

• The HCP datasource selector regex uses ".+$" here, but other dashboards use ".$" for Managed Prometheus HCP datasources. Aligning with ".$" keeps selector behavior consistent across dashboards.

        "regex": "^Managed_Prometheus_hcps-.+$",

observability/grafana-dashboards/sre/user-journey/sli-slo-metrics.json:2216

• Datasource variable regexes use ".+$" here, while the rest of the repo’s dashboards consistently use ".$" for Managed Prometheus datasources. Using ".$" avoids unexpectedly excluding a datasource if the suffix ever changes (and keeps this dashboard consistent with existing ones).

        "regex": "^Managed_Prometheus_services-.+$",

• Files reviewed: 1/1 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

A PromQL label_replace usage in the histogram panel uses an empty source-label argument, which can break query parsing/evaluation and prevent the panel from rendering.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 1/1 changed files
• Comments generated: 1
• Review effort level: Lite

[openshift-ci]

Simon Wiencki (@swiencki): The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-parallel	bdc4096	link	true	/test e2e-parallel

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.