feat(hcpctl): add Cosmos history to must-gather - #6834
Merged
Merged
Conversation
…ather Adds `hcpctl oc-adm-inspect`, a Kusto-backed equivalent of `oc adm inspect` that reconstructs a namespace's state at a point in time from telemetry instead of a live cluster: - Resources from kubernetesResourceSnapshots (the latest snapshot at or before the timestamp), excluding objects that were actually deleted by then (a Delete event, or a deletionTimestamp past its grace period). - Kubernetes events from kubernetesEvents. - Container logs from containerLogs, querying both the ServiceLogs and HostedControlPlaneLogs databases and merging them. Output mirrors oc adm inspect: the Namespace object at <ns>/<ns>.yaml, per-kind v1 List files at <group>/<resource>.yaml (e.g. core/pods.yaml), pods individually at pods/<pod>/<pod>.yaml, container logs at pods/<pod>/<container>/<container>/logs/current.log, and events at core/events.yaml, each with a leading "---" separator. Namespace inputs are validated as RFC 1123 labels and the writer refuses to write outside its output root. The time window uses --timestamp-min/--timestamp-max to match must-gather. Cluster IDs and namespaces are discovered by resource group / cluster id; hosted-cluster and control-plane namespaces are paired by their ocm- prefix relationship. When no management/service cluster is given, the active clusters are listed for the user. must-gather `query` runs oc-adm-inspect for every HCP cluster discovered in the resource group (skippable with --skip-oc-adm-inspect), writing under <output>/oc-adm-inspect, and reports success only when the run had no errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-test must-gather commands in the custom-link-tools debug links used a single global --subscription-id for every test, but e2e tests run in per-test subscriptions, so the command targeted the wrong subscription and its Kusto discovery matched nothing. Capture each test's subscription into the timing metadata and use it when building the per-test command: - Add a serialized SubscriptionID to SpecTimingMetadata (intentionally serialized, unlike the per-resource Resource.SubscriptionID, so the links can target the right subscription). - Stamp the test's subscription onto the timing metadata when it is written. - Surface it through timing.TimingInfo (LoadTestTimingInfo). - getPerTestMustGatherCommands uses the per-test subscription, falling back to the global --subscription-id flag for older metadata that lacks it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
openshift-ci
Bot
requested review from
Jan-Hendrik Boll (janboll) and
Miquel Simon (miquelsi)
September 4, 2026 15:44
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Must-gather’s Cosmos history generation currently runs unconditionally and can fail (or violate user intent) when --skip-custom-logs is used because the expected custom/ input path may not exist.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR enhances hcpctl must-gather with additional human-debugging artifacts by (1) generating a Git-history view of Cosmos resource snapshot data and (2) adding a Kusto-backed oc adm inspect equivalent to reconstruct namespace state/events/logs at a point in time. It also extends per-test timing metadata so generated must-gather links can target the correct per-test Azure subscription.
Changes:
- Add Cosmos snapshot → Git history generation into
hcpctl must-gather queryoutput (cosmosContent/). - Introduce
hcpctl oc-adm-inspectand a sharedpkg/ocadminspectimplementation with new builtin KQL templates. - Record per-test
SubscriptionIDin timing metadata and use it when generating per-test must-gather commands.
File summaries
| File | Description |
|---|---|
| tooling/utilitytypes/timing/types.go | Adds SubscriptionID to serialized test timing metadata for per-test must-gather targeting. |
| tooling/hcpctl/testdata/zz_fixture_TestTemplateDataOptions.yaml | Updates fixture to include new Namespace template variable. |
| tooling/hcpctl/testdata/zz_fixture_TestTemplateDataOptions_NoTruncation.yaml | Updates fixture to include new Namespace template variable. |
| tooling/hcpctl/pkg/ocadminspect/writer.go | Implements oc-adm-inspect-style filesystem writer with traversal guardrails. |
| tooling/hcpctl/pkg/ocadminspect/writer_test.go | Adds unit tests for writer layout and traversal rejection. |
| tooling/hcpctl/pkg/ocadminspect/row.go | Adds Kusto row→map conversion with dynamic-column JSON decoding. |
| tooling/hcpctl/pkg/ocadminspect/inspect.go | Implements Kusto-backed namespace reconstruction (resources/events/logs) and discovery helpers. |
| tooling/hcpctl/pkg/ocadminspect/inspect_test.go | Adds tests for namespace pairing and KQL template rendering expectations. |
| tooling/hcpctl/pkg/kusto/templates/builtin/queries.yaml | Registers new oc-adm-inspect builtin queries. |
| tooling/hcpctl/pkg/kusto/templates/builtin/oc-adm-inspect/resource_snapshots.kql.gotmpl | Adds KQL to reconstruct resource state at TimestampMax, excluding truly-deleted objects. |
| tooling/hcpctl/pkg/kusto/templates/builtin/oc-adm-inspect/namespaces.kql.gotmpl | Adds KQL to discover namespaces on a cluster (optionally filtered by cluster IDs). |
| tooling/hcpctl/pkg/kusto/templates/builtin/oc-adm-inspect/events.kql.gotmpl | Adds KQL to fetch Kubernetes events for a namespace in the window. |
| tooling/hcpctl/pkg/kusto/templates/builtin/oc-adm-inspect/container_logs.kql.gotmpl | Adds KQL to fetch ServiceLogs container logs for a namespace. |
| tooling/hcpctl/pkg/kusto/templates/builtin/oc-adm-inspect/container_logs_hcp.kql.gotmpl | Adds KQL to fetch HostedControlPlaneLogs container logs by namespace. |
| tooling/hcpctl/pkg/kusto/templates/builtin/oc-adm-inspect/active_clusters.kql.gotmpl | Adds KQL to discover active cluster names to guide users when cluster isn’t specified. |
| tooling/hcpctl/pkg/kusto/query.go | Adds Namespace template field and WithNamespace option with KQL escaping. |
| tooling/hcpctl/pkg/kusto/query_definitions.go | Adds query-type constants for oc-adm-inspect query categorization. |
| tooling/hcpctl/main.go | Registers the new hcpctl oc-adm-inspect command. |
| tooling/hcpctl/cmd/oc-adm-inspect/options.go | Adds CLI options/validation/completion for oc-adm-inspect (namespace normalization + safety checks). |
| tooling/hcpctl/cmd/oc-adm-inspect/options_test.go | Adds tests for namespace normalization and validation behavior. |
| tooling/hcpctl/cmd/oc-adm-inspect/cmd.go | Adds command entrypoint and output messaging for discovery vs. inspect execution. |
| tooling/hcpctl/cmd/must-gather/query_options.go | Adds --skip-oc-adm-inspect flag to control new must-gather behavior. |
| tooling/hcpctl/cmd/must-gather/query_cmd.go | Runs Cosmos history generation and (optionally) oc-adm-inspect after log gathering. |
| tooling/hcpctl/cmd/must-gather/query_cmd_test.go | Adds tests for Cosmos history generation (including empty-input behavior). |
| tooling/hcpctl/cmd/must-gather/cmd.go | Adds constant for oc-adm-inspect output directory name. |
| tooling/hcpctl/cmd/datadump-to-git/options.go | Exposes options type for programmatic Cosmos history generation from must-gather. |
| tooling/hcpctl/cmd/datadump-to-git/cmd.go | Adds parsing for must-gather Cosmos snapshot JSONL rows into datadump entries. |
| tooling/hcpctl/cmd/datadump-to-git/cmd_test.go | Adds unit tests for Cosmos snapshot row parsing and timestamp derivation. |
| test/util/timing/timewindow.go | Plumbs SubscriptionID from timing metadata into loaded timing info. |
| test/util/framework/per_test_framework.go | Records per-test SubscriptionID into timing metadata at test completion. |
| test/cmd/aro-hcp-tests/custom-link-tools/options.go | Uses per-test subscription ID when generating per-test must-gather commands (with fallback). |
Review details
- Files reviewed: 31/31 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+74
to
+78
| logger.Info("gathering must-gather logs", "output", opts.OutputPath) | ||
| gatherErr := gatherer.GatherLogs(ctx) | ||
|
|
||
| cosmosContentErr := generateCosmosContent(ctx, opts.OutputPath) | ||
|
|
Comment on lines
+39
to
46
| type CosmosSnapshotToGitRepoOptions struct { | ||
| LogPath string | ||
| OutputDir string | ||
| } | ||
|
|
||
| func defaultOptions() *rawOptions { | ||
| return &rawOptions{} | ||
| func DefaultCosmosSnapshotToGitRepoOptions() *CosmosSnapshotToGitRepoOptions { | ||
| return &CosmosSnapshotToGitRepoOptions{} | ||
| } |
Contributor
|
/lgtm |
Contributor
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: deads2k, stevekuznetsov The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes it more convenient for humans to debug. Might eventually choose to do similar for kubernetesEvents, but not starting there.