Skip to content

cnv-must-gather: add promtool backfill validation to test-incident CI step - #83036

Open
tiraboschi wants to merge 1 commit into
openshift:mainfrom
tiraboschi:cnv_mustgather_incident_backfill
Open

cnv-must-gather: add promtool backfill validation to test-incident CI step#83036
tiraboschi wants to merge 1 commit into
openshift:mainfrom
tiraboschi:cnv_mustgather_incident_backfill

Conversation

@tiraboschi

@tiraboschi tiraboschi commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Validate that the incident metrics file produced by --vm-incident is actually importable into Prometheus. Extract promtool from the cluster's own monitoring stack image and run tsdb create-blocks-from openmetrics against the collected incident-metrics.txt. Fails the CI step if the OpenMetrics format is invalid (e.g. duplicate labels, malformed lines).

Summary by CodeRabbit

  • Updates Azure and GCP cnv-must-gather incident CI tests.
  • Extracts promtool from the cluster monitoring stack image.
  • Validates incident-metrics.txt with tsdb create-blocks-from openmetrics.
  • Fails the test and saves validation output when the OpenMetrics format is invalid.
  • Skips validation with a warning when incident-metrics.txt is missing.
  • Removes temporary extraction and backfill directories after validation.
  • Adds a one-minute grace period to both incident test steps.

@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: c544e2d7-be31-4feb-96cb-a41ab7796780

📥 Commits

Reviewing files that changed from the base of the PR and between d3c6320 and 3963076.

📒 Files selected for processing (1)
  • ci-operator/config/kubevirt/must-gather/kubevirt-must-gather-main.yaml
🚧 Files skipped from review as they are similar to previous changes (1)
  • ci-operator/config/kubevirt/must-gather/kubevirt-must-gather-main.yaml

Walkthrough

Azure and GCP incident tests now validate collected incident-metrics.txt with promtool. Missing metrics produce warnings and skip validation. Failed validation saves raw output and exits. Temporary resources are removed, and both tests add a one-minute grace period.

Changes

Incident metrics validation

Layer / File(s) Summary
Incident test metrics validation
ci-operator/config/kubevirt/must-gather/kubevirt-must-gather-main.yaml
Azure and GCP incident tests extract promtool, validate OpenMetrics backfill, clean temporary resources, warn when metrics are missing, save failed validation output, exit on validation errors, and add a one-minute grace period.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: orenc1, nunnatsa


Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error, 1 warning)

Check name Status Explanation Resolution
No-Sensitive-Data-In-Logs ❌ Error The failure artifact stores promtool diagnostics from cluster metrics; its sed rule redacts only ="..." values, so unquoted tokens, hostnames, or secrets can remain exposed. Do not upload raw parser diagnostics. Store only a fixed error summary, or apply complete structured redaction to all input-derived text before writing the artifact.
Microshift Test Compatibility ⚠️ Warning The new Azure and GCP incident test steps query openshift-monitoring/prometheus-k8s, a MicroShift-unsupported monitoring component, without a MicroShift skip. Add [Skipped:MicroShift] or a runtime IsMicroShiftCluster()+g.Skip() guard. Otherwise run the prescribed MicroShift conformance payload job.
✅ Passed checks (13 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the added promtool validation in the cnv-must-gather incident CI step.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The PR adds no Ginkgo title. Both incident CI steps use the static name test-incident; timestamps and identifiers remain in command bodies.
Test Structure And Quality ✅ Passed The PR changes only CI YAML shell commands; it adds no Ginkgo It blocks, assertions, waits, or cluster resource creation, and it cleans temporary paths with an EXIT trap.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The PR changes only CI YAML shell commands for existing incident tests; it adds no Ginkgo test declarations or multi-node/HA assumptions.
Topology-Aware Scheduling Compatibility ✅ Passed The only changed file is a CI YAML; its additions are shell validation and grace periods, with no deployment, controller, or scheduling constraints.
Ote Binary Stdout Contract ✅ Passed The PR modifies only a CI configuration YAML file, adding shell commands to CI test steps. It contains no Go source code changes affecting OTE binary entry points (main, init, TestMain, suite setup...
Ipv6 And Disconnected Network Test Compatibility ✅ Passed This PR modifies a CI operator configuration file, not Ginkgo e2e tests. The custom check applies only when new Ginkgo e2e tests are added; no Go test code changes are present.
No-Weak-Crypto ✅ Passed The PR adds only YAML and shell validation logic; scans of all added lines found no MD5, SHA1, DES, RC4, 3DES, Blowfish, ECB, custom crypto, or secret comparisons.
Container-Privileges ✅ Passed The PR adds CI shell commands and grace periods only; the YAML diff contains no privileged, hostPID, hostNetwork, hostIPC, SYS_ADMIN, or container security settings.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci

openshift-ci Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tiraboschi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 6, 2026
@openshift-ci
openshift-ci Bot requested review from nunnatsa and orenc1 August 6, 2026 12:59

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@ci-operator/config/kubevirt/must-gather/kubevirt-must-gather-main.yaml`:
- Around line 128-133: Add EXIT traps for cleanup in both validation blocks:
ci-operator/config/kubevirt/must-gather/kubevirt-must-gather-main.yaml lines
128-133 should clean the Azure temporary directories, and lines 274-279 should
clean the GCP temporary directories. Ensure cleanup runs on validation failure
as well as success, while retaining the existing successful-validation flow.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 0a6b75f6-c61e-40a9-95e5-f6fbe117c458

📥 Commits

Reviewing files that changed from the base of the PR and between 3abed54 and 029ff6f.

📒 Files selected for processing (1)
  • ci-operator/config/kubevirt/must-gather/kubevirt-must-gather-main.yaml

Comment thread ci-operator/config/kubevirt/must-gather/kubevirt-must-gather-main.yaml Outdated
@tiraboschi
tiraboschi force-pushed the cnv_mustgather_incident_backfill branch from 029ff6f to 086c1d3 Compare August 6, 2026 13:09
@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@tiraboschi, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

could not load configuration from candidate revision of release repo: failed to load ci-operator configuration from release repo: invalid ci-operator config: configuration has 2 errors:

 * test `test-incident` has `commands` containing `trap` command, but test step is missing grace_period
 * test `test-incident` has `commands` containing `trap` command, but test step is missing grace_period

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@tiraboschi
tiraboschi force-pushed the cnv_mustgather_incident_backfill branch from 086c1d3 to 9e6a6b5 Compare August 6, 2026 13:19
@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

… step

Validate that the incident metrics file produced by --vm-incident is
actually importable into Prometheus. Extract promtool from the cluster's
own monitoring stack image and run tsdb create-blocks-from openmetrics
against the collected incident-metrics.txt. Fails the CI step if the
OpenMetrics format is invalid (e.g. duplicate labels, malformed lines).

Signed-off-by: Simone Tiraboschi <stirabos@redhat.com>
@tiraboschi
tiraboschi force-pushed the cnv_mustgather_incident_backfill branch from 9e6a6b5 to 3963076 Compare August 6, 2026 13:33
@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@tiraboschi: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-kubevirt-must-gather-main-kubevirt-must-gather-e2e-azure kubevirt/must-gather presubmit Ci-operator config changed
pull-ci-kubevirt-must-gather-main-kubevirt-must-gather-e2e-gcp kubevirt/must-gather presubmit Ci-operator config changed
Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@openshift-ci

openshift-ci Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

@tiraboschi: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@tiraboschi

Copy link
Copy Markdown
Contributor Author

/pj-rehearse

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@tiraboschi: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant