test(bdd): prepare self-managed secrets safely

tests/bdd/AGENTS.md

@@ -26,9 +26,9 @@ assertion as the escape hatch for uncommon or command-specific behavior.
   `CommandCache`, `Suite`. Step handlers depend on these; nothing else
   does.
 - `dsl/` owns pure helpers: `${VAR}` interpolation, dotted-path YAML
-  upsert and read, YAML subtree match/contain, kubectl manifest
-  builders, JSON row matching. Every helper is unit-testable in
-  isolation. No I/O coordination, no Godog dependency.
+  upsert and read, YAML subtree match/contain, self-managed secrets
+  rendering, kubectl manifest builders, JSON row matching. Every helper
+  is unit-testable in isolation. No I/O coordination, no Godog dependency.
 - `steps/` owns Godog step handlers and `ScenarioContext`. Each
   handler is one or two lines plus a delegate to a `dsl` helper or
   `Suite.Runner`.
@@ -45,9 +45,9 @@ logic into `dsl/`.
   a bare `$word` is left literal. Implementations must not use
   `os.ExpandEnv`. Expansion lives in `dsl.Interpolate`.
 - File-mutating steps (`I copy the file`, `I update yaml file`,
-  `I substitute`) snapshot the destination through `Suite.Ledger`
-  before the first write. Suite teardown restores every snapshotted
-  path.
+  `I prepare self-managed secrets file`, `I substitute a block`)
+  snapshot the destination through `Suite.Ledger` before the first write.
+  Suite teardown restores every snapshotted path.
 - `Given command has succeeded:` keys on the fully resolved command
   text. Two scenarios whose pre-interpolation text matches but whose
   env vars differ must miss the cache. The cache lives in

tests/bdd/PLAN.md

@@ -10,7 +10,7 @@ domain-specific validation logic.
 
 The vocabulary is restricted to four categories:
 
-1. File operations: copy, edit YAML, substitute strings.
+1. File operations: copy, edit YAML, substitute blocks, prepare secrets.
 2. Environment preconditions: env vars set, files exist, infrastructure
    reachable.
 3. Command execution: exec a shell command and capture exit code, stdout,
@@ -105,7 +105,7 @@ refactor in every consumer; that is a feature.
 |------|-------|
 | `And I copy the file {string} to {string}` | Both paths are repo-relative. |
 | `And I update yaml file {string} with keys:` (two-column table of dotted-path and value) | Path supports dotted notation and `[n]` indices (e.g. `global.imagePullSecrets[0].name`). Missing intermediate maps and missing list indices are upserted: writing `global.imagePullSecrets[0].name` against a file that has neither `global.imagePullSecrets` nor any list entry creates both. Existing scalars at intermediate positions cause the step to fail rather than silently overwrite a non-map. Value cells expand `${VAR}` from `os.Environ`. |
-| `And I substitute {string} in file {string} with base64 of {string}` | Used for credential rendering; the third arg expands `${VAR}` then base64-encodes. The handler never logs the substituted value. |
+| `And I prepare self-managed secrets file {string} from template {string} using the current NGC registry credential` | The destination and template are explicit repo-relative paths with `${VAR}` interpolation. Replaces the template's registry credential placeholder with base64 of the current `$oauthtoken:<NGC_API_KEY>` credential and writes the destination with mode `0600`. The destination is ledger-backed, and secret material never enters Gherkin, command logs, or failure messages. |
 | `And I substitute a block in file {string}:` (docstring) | The docstring contains an old block and replacement block separated by exactly one `---` line. `${VAR}` interpolation applies before an exact, ledger-backed replacement. Missing or malformed old blocks fail. |
 
 ### Command execution (When)
@@ -213,7 +213,8 @@ contract verified in `src/clis/nvcf-cli/cmd/`):
 
 Every step that writes into a path under the repo working tree
 (`I copy the file ... to ...`, `I update yaml file ...`,
-`I substitute ... in file ...`) registers that path with the runner's
+`I prepare self-managed secrets file ...`, `I substitute a block ...`)
+registers that path with the runner's
 restoration ledger:
 
 - Before the first write, the runner snapshots the file (exists/not,
@@ -448,9 +449,14 @@ const (
 // names the first path that differed.
 func MatchYAMLSubtree(filePath, keyPath, expectedYAML string, mode MatchMode) error
 
+// RenderSelfManagedSecrets replaces the secrets-template registry credential
+// placeholder with base64 of `$oauthtoken:<NGC_API_KEY>`. It fails without
+// returning raw or encoded credential material when the key or placeholder is
+// missing.
+func RenderSelfManagedSecrets(template []byte, apiKey string) ([]byte, error)
+
 // SubstituteFile replaces every occurrence of placeholder with
-// replacement in the named file. Used for credential rendering.
-// Never logs placeholder or replacement.
+// replacement in the named file. Never logs placeholder or replacement.
 func SubstituteFile(path, placeholder, replacement string) error
 
 // JSONContainsRows parses raw as a JSON array of objects, and for each

tests/bdd/dsl/secrets.go

@@ -0,0 +1,41 @@
+/*
+SPDX-FileCopyrightText: Copyright (c) NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package dsl
+
+import (
+	"bytes"
+	"encoding/base64"
+	"fmt"
+)
+
+const selfManagedRegistryCredentialPlaceholder = "REPLACE_WITH_BASE64_DOCKER_CREDENTIAL"
+
+// RenderSelfManagedSecrets replaces the registry credential placeholder with
+// base64 of the Docker username and current NGC API key. Errors never include
+// the raw or encoded credential.
+func RenderSelfManagedSecrets(template []byte, apiKey string) ([]byte, error) {
+	if apiKey == "" {
+		return nil, fmt.Errorf("NGC_API_KEY is not set")
+	}
+	placeholder := []byte(selfManagedRegistryCredentialPlaceholder)
+	if !bytes.Contains(template, placeholder) {
+		return nil, fmt.Errorf("self-managed secrets template is missing the registry credential placeholder")
+	}
+	credential := base64.StdEncoding.EncodeToString([]byte("$oauthtoken:" + apiKey))
+	return bytes.ReplaceAll(template, placeholder, []byte(credential)), nil
+}

tests/bdd/dsl/secrets_test.go

@@ -0,0 +1,63 @@
+/*
+SPDX-FileCopyrightText: Copyright (c) NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package dsl
+
+import (
+	"encoding/base64"
+	"strings"
+	"testing"
+)
+
+func TestRenderSelfManagedSecretsUsesDockerCredentialFormat(t *testing.T) {
+	apiKey := "test-api-key"
+	template := []byte("first: " + selfManagedRegistryCredentialPlaceholder + "\nsecond: " + selfManagedRegistryCredentialPlaceholder + "\n")
+
+	got, err := RenderSelfManagedSecrets(template, apiKey)
+	if err != nil {
+		t.Fatalf("render: %v", err)
+	}
+	wantCredential := base64.StdEncoding.EncodeToString([]byte("$oauthtoken:" + apiKey))
+	if strings.Count(string(got), wantCredential) != 2 {
+		t.Fatalf("rendered credential count = %d, want 2", strings.Count(string(got), wantCredential))
+	}
+	if strings.Contains(string(got), apiKey) {
+		t.Fatal("raw API key leaked into rendered secrets")
+	}
+}
+
+func TestRenderSelfManagedSecretsRejectsMissingAPIKey(t *testing.T) {
+	_, err := RenderSelfManagedSecrets([]byte(selfManagedRegistryCredentialPlaceholder), "")
+	if err == nil || !strings.Contains(err.Error(), "NGC_API_KEY is not set") {
+		t.Fatalf("error = %v, want missing-key error", err)
+	}
+}
+
+func TestRenderSelfManagedSecretsFailureHidesCredentialMaterial(t *testing.T) {
+	apiKey := "sensitive-test-api-key"
+	encoded := base64.StdEncoding.EncodeToString([]byte("$oauthtoken:" + apiKey))
+
+	_, err := RenderSelfManagedSecrets([]byte("registryCredential: missing\n"), apiKey)
+	if err == nil {
+		t.Fatal("expected missing-placeholder error")
+	}
+	for _, secret := range []string{apiKey, encoded} {
+		if strings.Contains(err.Error(), secret) {
+			t.Fatalf("error leaked credential material: %v", err)
+		}
+	}
+}

tests/bdd/features/multi-cluster-eks-helmfile.feature

@@ -73,8 +73,7 @@ Feature: Install a multi-cluster NVCF stack across two pre-provisioned EKS clust
       bash -c 'set -eo pipefail; printf %s "$NGC_API_KEY" | helm registry login nvcr.io --username "\$oauthtoken" --password-stdin'
       """
     # Create NGC dockerconfig registry credentials.
-    And I copy the file "deploy/stacks/self-managed/secrets/secrets.yaml.template" to "deploy/stacks/self-managed/secrets/eks-bdd-multi-secrets.yaml"
-    And I substitute "REPLACE_WITH_BASE64_DOCKER_CREDENTIAL" in file "deploy/stacks/self-managed/secrets/eks-bdd-multi-secrets.yaml" with base64 of "$oauthtoken:${NGC_API_KEY}"
+    And I prepare self-managed secrets file "deploy/stacks/self-managed/secrets/eks-bdd-multi-secrets.yaml" from template "deploy/stacks/self-managed/secrets/secrets.yaml.template" using the current NGC registry credential
     # Both clusters must be reachable before we start.
     And I run command "kubectl --context ${EKS_CONTEXT} get nodes -o name"
     And the command exit code should be 0
@@ -412,6 +411,8 @@ Feature: Install a multi-cluster NVCF stack across two pre-provisioned EKS clust
       Then the command exit code should be 0
       And the command output should contain "bdd-echo"
 
+    # Failing until GitHub issue #1098 is resolved and the fix from GitHub
+    # issue #1032 is consumed by the self-managed stack.
     @nvct-task-api
     Scenario: User launches an NVCT task on the compute cluster and waits for completion
       Given command has succeeded:

tests/bdd/features/multi-cluster-helmfile.feature

@@ -50,8 +50,7 @@ Feature: Install a local multi-cluster NVCF stack with Helmfile
         | global.helm.sources.repository  | ${SAMPLE_NGC_ORG}/${SAMPLE_NGC_TEAM} |
         | global.image.repository         | ${SAMPLE_NGC_ORG}/${SAMPLE_NGC_TEAM} |
         | observability.profile           | disabled                             |
-      And I copy the file "deploy/stacks/self-managed/secrets/secrets.yaml.template" to "deploy/stacks/self-managed/secrets/local-bdd-secrets.yaml"
-      And I substitute "REPLACE_WITH_BASE64_DOCKER_CREDENTIAL" in file "deploy/stacks/self-managed/secrets/local-bdd-secrets.yaml" with base64 of "$oauthtoken:${NGC_API_KEY}"
+      And I prepare self-managed secrets file "deploy/stacks/self-managed/secrets/local-bdd-secrets.yaml" from template "deploy/stacks/self-managed/secrets/secrets.yaml.template" using the current NGC registry credential
       # Conflict precheck: single-cluster ncp-local's k3d serverlb
       # claims 0.0.0.0:8080/8443/10081, and ncp-local-cp also
       # needs NATS on 4222 plus the worker callback port 10086.
@@ -235,6 +234,8 @@ Feature: Install a local multi-cluster NVCF stack with Helmfile
     # This scenario intentionally has no Background. It depends on the
     # earlier control-plane install and NVCA registration scenarios in
     # this feature run, and is not a standalone tag target.
+    # Failing until GitHub issue #1098 is resolved and the fix from GitHub
+    # issue #1032 is consumed by the self-managed stack.
     @nvct-task-api
     Scenario: Operator launches an NVCT task and waits for it to complete
       When I run command:

tests/bdd/features/multi-cluster-up.feature

@@ -20,17 +20,14 @@ Feature: Bring up a local multi-cluster NVCF stack with the CLI
         | NGC_API_KEY     |
         | SAMPLE_NGC_ORG  |
         | SAMPLE_NGC_TEAM |
-      # self-hosted install --env local reads operator-authored local
-      # secrets files from both split stacks:
-      # deploy/stacks/self-managed/secrets/local-secrets.yaml (control
-      # plane). Only secrets.yaml.template is tracked in each
-      # stack. Author both files from the canonical templates before
-      # running install/register. Ledger snapshots whatever
+      # self-hosted install --env local reads the operator-authored
+      # control-plane secrets file. Only secrets.yaml.template is tracked.
+      # Prepare local-secrets.yaml from that template before running
+      # install/register. Ledger snapshots whatever
       # local-secrets.yaml state existed before the first write (its
       # prior contents or absence) and restores or removes it at suite
       # teardown.
-      And I copy the file "deploy/stacks/self-managed/secrets/secrets.yaml.template" to "deploy/stacks/self-managed/secrets/local-secrets.yaml"
-      And I substitute "REPLACE_WITH_BASE64_DOCKER_CREDENTIAL" in file "deploy/stacks/self-managed/secrets/local-secrets.yaml" with base64 of "$oauthtoken:${NGC_API_KEY}"
+      And I prepare self-managed secrets file "deploy/stacks/self-managed/secrets/local-secrets.yaml" from template "deploy/stacks/self-managed/secrets/secrets.yaml.template" using the current NGC registry credential
       # --env local also reads operator-authored environment values from
       # both split stacks: deploy/stacks/<stack>/environments/local.yaml.
       # Neither file is tracked, so author both from the BDD multi-cluster

tests/bdd/features/observability-all.feature

@@ -42,8 +42,7 @@ Feature: Install local Helmfile observability for both planes
       | global.image.repository                                       | ${SAMPLE_NGC_ORG}/${SAMPLE_NGC_TEAM}                              |
       | global.nvcaOperator.selfManaged.otelCollector.imageRepository | nvcr.io/${SAMPLE_NGC_ORG}/${SAMPLE_NGC_TEAM}/nvcf-otel-collector |
       | observability.profile                                         | all                                                               |
-    And I copy the file "deploy/stacks/self-managed/secrets/secrets.yaml.template" to "deploy/stacks/self-managed/secrets/local-bdd-observability-all-secrets.yaml"
-    And I substitute "REPLACE_WITH_BASE64_DOCKER_CREDENTIAL" in file "deploy/stacks/self-managed/secrets/local-bdd-observability-all-secrets.yaml" with base64 of "$oauthtoken:${NGC_API_KEY}"
+    And I prepare self-managed secrets file "deploy/stacks/self-managed/secrets/local-bdd-observability-all-secrets.yaml" from template "deploy/stacks/self-managed/secrets/secrets.yaml.template" using the current NGC registry credential
     # Conflict precheck: the split topology claims host ports used by the
     # single-cluster topology. From the repository root, run
     # `make -C tools/ncp-local-cluster destroy-all-ncp-local SHELL=/bin/bash`

tests/bdd/features/observability-compute.feature

@@ -44,8 +44,7 @@ Feature: Install local Helmfile observability with the compute profile
       | global.image.repository                                       | ${SAMPLE_NGC_ORG}/${SAMPLE_NGC_TEAM}                              |
       | global.nvcaOperator.selfManaged.otelCollector.imageRepository | nvcr.io/${SAMPLE_NGC_ORG}/${SAMPLE_NGC_TEAM}/nvcf-otel-collector |
       | observability.profile                                         | compute                                                           |
-    And I copy the file "deploy/stacks/self-managed/secrets/secrets.yaml.template" to "deploy/stacks/self-managed/secrets/local-bdd-observability-compute-secrets.yaml"
-    And I substitute "REPLACE_WITH_BASE64_DOCKER_CREDENTIAL" in file "deploy/stacks/self-managed/secrets/local-bdd-observability-compute-secrets.yaml" with base64 of "$oauthtoken:${NGC_API_KEY}"
+    And I prepare self-managed secrets file "deploy/stacks/self-managed/secrets/local-bdd-observability-compute-secrets.yaml" from template "deploy/stacks/self-managed/secrets/secrets.yaml.template" using the current NGC registry credential
     # Conflict precheck: single-cluster ncp-local claims host ports used by the
     # split topology. From the repository root, run
     # `make -C tools/ncp-local-cluster destroy CLUSTER_NAME=ncp-local`

tests/bdd/features/observability-control.feature

@@ -31,8 +31,7 @@ Feature: Install local Helmfile observability with the control profile
       | global.helm.sources.repository  | ${SAMPLE_NGC_ORG}/${SAMPLE_NGC_TEAM} |
       | global.image.repository         | ${SAMPLE_NGC_ORG}/${SAMPLE_NGC_TEAM} |
       | observability.profile           | control                              |
-    And I copy the file "deploy/stacks/self-managed/secrets/secrets.yaml.template" to "deploy/stacks/self-managed/secrets/local-bdd-observability-control-secrets.yaml"
-    And I substitute "REPLACE_WITH_BASE64_DOCKER_CREDENTIAL" in file "deploy/stacks/self-managed/secrets/local-bdd-observability-control-secrets.yaml" with base64 of "$oauthtoken:${NGC_API_KEY}"
+    And I prepare self-managed secrets file "deploy/stacks/self-managed/secrets/local-bdd-observability-control-secrets.yaml" from template "deploy/stacks/self-managed/secrets/secrets.yaml.template" using the current NGC registry credential
     # Conflict precheck: ncp-local-cp claims host ports that overlap with this
     # single-cluster topology. From tools/ncp-local-cluster, run
     # `make destroy CLUSTER_NAME=ncp-local-cp` before retrying.

tests/bdd/features/observability-disabled.feature

@@ -24,8 +24,7 @@ Feature: Render local Helmfile stacks with observability disabled
       | global.helm.sources.repository  | ${SAMPLE_NGC_ORG}/${SAMPLE_NGC_TEAM} |
       | global.image.repository         | ${SAMPLE_NGC_ORG}/${SAMPLE_NGC_TEAM} |
       | observability.profile           | disabled                             |
-    And I copy the file "deploy/stacks/self-managed/secrets/secrets.yaml.template" to "deploy/stacks/self-managed/secrets/local-bdd-observability-disabled-secrets.yaml"
-    And I substitute "REPLACE_WITH_BASE64_DOCKER_CREDENTIAL" in file "deploy/stacks/self-managed/secrets/local-bdd-observability-disabled-secrets.yaml" with base64 of "$oauthtoken:${NGC_API_KEY}"
+    And I prepare self-managed secrets file "deploy/stacks/self-managed/secrets/local-bdd-observability-disabled-secrets.yaml" from template "deploy/stacks/self-managed/secrets/secrets.yaml.template" using the current NGC registry credential
     # Create the compute-plane stack environment used by the worker render.
     And I copy the file "tests/bdd/fixtures/nvcf-compute-plane-local-bdd.yaml" to "deploy/stacks/nvcf-compute-plane/environments/local-bdd-observability-disabled.yaml"
     And I update yaml file "deploy/stacks/nvcf-compute-plane/environments/local-bdd-observability-disabled.yaml" with keys:

tests/bdd/features/single-cluster-eks-helmfile.feature

@@ -71,8 +71,7 @@ Feature: Install a single-cluster NVCF stack on a pre-provisioned EKS cluster wi
       bash -c 'set -eo pipefail; printf %s "$NGC_API_KEY" | helm registry login nvcr.io --username "\$oauthtoken" --password-stdin'
       """
     # Create NGC dockerconfig registry credentials
-    And I copy the file "deploy/stacks/self-managed/secrets/secrets.yaml.template" to "deploy/stacks/self-managed/secrets/eks-bdd-secrets.yaml"
-    And I substitute "REPLACE_WITH_BASE64_DOCKER_CREDENTIAL" in file "deploy/stacks/self-managed/secrets/eks-bdd-secrets.yaml" with base64 of "$oauthtoken:${NGC_API_KEY}"
+    And I prepare self-managed secrets file "deploy/stacks/self-managed/secrets/eks-bdd-secrets.yaml" from template "deploy/stacks/self-managed/secrets/secrets.yaml.template" using the current NGC registry credential
     And I run command "kubectl --context ${EKS_CONTEXT} get nodes -o name"
     And the command exit code should be 0