Skip to content

fix(bdd): use exported profile trust for PKI #1266

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mikeyrcamp merged 1 commit into from
Aug 27, 2026
Merged

fix(bdd): use exported profile trust for PKI #1266

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 22 additions & 16 deletions tests/bdd/features/single-cluster-helmfile-llm-pki.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,10 @@ Feature: Install a local single-cluster NVCF stack with PKI-secured LLM transpor
so that an LLM function answers invocations over a QUIC tunnel whose
trust chain is issued by the stack's own PKI.

# Owns its own Helmfile environment (local-bdd-pki) so the default
# fixtures and the non-PKI features stay unchanged; PKI enablement
# is an install-time value. The trust bundle (OpenBao root CA plus
# nvcf-trust-bundle-v1 fingerprint) only exists after the control
# plane is up, so a helper script writes it into the compute
# environment between install and register. That rewrite drops
# stargateQUICInsecure: the tunnel runs in secure mode.
# Owns its own Helmfile environment (local-bdd-pki) for install-time PKI
# values. The exported profile carries the OpenBao root CA and fingerprint
# into registration values. The shared compute fixture uses secure QUIC so
# the registered bundle trust remains active.

Rule: Helmfile installs the control plane with the LLM PKI addon

Expand Down Expand Up @@ -124,6 +121,18 @@ Feature: Install a local single-cluster NVCF stack with PKI-secured LLM transpor
"""
Then the command exit code should be 0
And file "deploy/stacks/self-managed/out/control-plane-profile.yaml" should exist
And yaml file "deploy/stacks/self-managed/out/control-plane-profile.yaml" should contain:
"""
managementTls:
trustMode: bundle
transportTls:
trustMode: bundle
"""
And yaml file "deploy/stacks/self-managed/out/control-plane-profile.yaml" should have non-empty keys:
| key |
| managementTls.caBundlePem |
| transportTls.trustBundleFingerprint |
| transportTls.trustBundlePem |

# The profile carries endpoints and trust, not credentials. Initialize
# the harness-isolated, config-scoped CLI state before compute-plane
Expand Down Expand Up @@ -154,15 +163,6 @@ Feature: Install a local single-cluster NVCF stack with PKI-secured LLM transpor
Then the command exit code should be 0
And file "deploy/stacks/nvcf-compute-plane/registration/ncp-local-register-values.yaml" should exist

# Fetch the root CA from OpenBao and write the transportTLS
# bundle block into the compute environment.
When I run command:
"""
tests/bdd/scripts/write-transport-trust-env.sh deploy/stacks/nvcf-compute-plane/environments/local-bdd-pki.yaml k3d-ncp-local
"""
Then the command exit code should be 0
And the command output should contain "fingerprint sha256:"

When I run command:
"""
make -C deploy/stacks/nvcf-compute-plane install CLUSTER_NAME=ncp-local HELMFILE_ENV=local-bdd-pki COMPUTE_KUBE_CONTEXT=k3d-ncp-local NVCF_CLI=${NVCF_CLI}
Expand All @@ -173,6 +173,12 @@ Feature: Install a local single-cluster NVCF stack with PKI-secured LLM transpor
| name | namespace |
| nvca-operator | nvca-operator |

When I run command "helm get values nvca-operator --namespace nvca-operator --kube-context k3d-ncp-local -o yaml"
Then the command exit code should be 0
And the command output should contain "stargateQUICInsecure: false"
And the command output should contain "trustMode: bundle"
And the command output should contain "trustBundleFingerprint: sha256:"

When I run command "kubectl --context k3d-ncp-local rollout status deployment/nvca-operator -n nvca-operator --timeout=10m"
Then the command exit code should be 0

Expand Down
222 changes: 0 additions & 222 deletions tests/bdd/fixtures_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,6 @@ import (
"testing"

"gopkg.in/yaml.v3"

"nvcf-bdd/dsl"
)

func TestSelfManagedOpenBaoWebhookDefaultsToIgnore(t *testing.T) {
Expand Down Expand Up @@ -292,226 +290,6 @@ func TestNVCTTaskSmokeUsesTaskSimpleSample(t *testing.T) {
}
}

func TestWriteTransportTrustEnvPreservesExistingAgentConfig(t *testing.T) {
const (
certificatePEM = `-----BEGIN CERTIFICATE-----
MIIEMjCCAxqgAwIBAgIUIZMDB4PqZ1PUCS8TIX7uqZXSDZ4wDQYJKoZIhvcNAQEL
BQAwgZAxCzAJBgNVBAYTAlVTMRIwEAYDVQQIDAlUZXN0U3RhdGUxETAPBgNVBAcM
CFRlc3RDaXR5MTQwMgYDVQQKDCtURVNUIENFUlRJRklDQVRFIC0gRE8gTk9UIFVT
RSBJTiBQUk9EVUNUSU9OMRAwDgYDVQQLDAdUZXN0aW5nMRIwEAYDVQQDDAlsb2Nh
bGhvc3QwHhcNMjYwMzA5MTMwMDM1WhcNMzYwMzA2MTMwMDM1WjCBkDELMAkGA1UE
BhMCVVMxEjAQBgNVBAgMCVRlc3RTdGF0ZTERMA8GA1UEBwwIVGVzdENpdHkxNDAy
BgNVBAoMK1RFU1QgQ0VSVElGSUNBVEUgLSBETyBOT1QgVVNFIElOIFBST0RVQ1RJ
T04xEDAOBgNVBAsMB1Rlc3RpbmcxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBAJk2OOKYkbEwWWTcfk+ovgS5gMpXQq3U
83ySD3tFczbT6bjXsBzJWyLa1s5/+IRbYTR0N2spmeNc4yrWRAUIzmRVIMADGHI5
LlN8emM8PzskrYK+MPh3kqZlNCVbjcdXoanFbPz8PJrbE4ohOcFj/Sb5dixjAvu+
26wDFdm06pw2cggumbpyKuKgFEFR1UHucUPfCv+X8TkKqRaDDIAf3FBQThBzJYzJ
XJJ3RxAAQzz6XJsqHit4Oc1tu3Rv6rKI7h6c2jYYToZHl1IhenQZX6k/rQnHg2Li
p2kOnIemtA9ccLYT7Eek3I5i1wjviujTbuKHA0qFOAkMIH5QZGdAh3kCAwEAAaOB
gTB/MB0GA1UdDgQWBBRmkXwkBK2g5adgUXQbtGf5Y9Vo5jAfBgNVHSMEGDAWgBRm
kXwkBK2g5adgUXQbtGf5Y9Vo5jAPBgNVHRMBAf8EBTADAQH/MCwGA1UdEQQlMCOC
CWxvY2FsaG9zdIcEfwAAAYcQAAAAAAAAAAAAAAAAAAAAATANBgkqhkiG9w0BAQsF
AAOCAQEAOxTBY4FzafF0QlfKyeRDgcJS+xSVUMBHW5Lnc/mWwtHEUvuhFSvGqEim
irnaoqW9ijUs70ful+VIUfucTLhQSxIEBiD6Wewxxe91I3sClW4hbrcVL7kzxLK5
N1UmMOeGIpB+DY/sy6HNF3vNEqqoHU6sZQasZGE/HbFPG8xvwwZlxk0hvZNTAQ9G
C3thGnRlMMjPOsnxnKeledidtW+NZXrKbcikgbBp/kQhwkqlHxpPkKdrUAn88YkP
+/zy1JTpb0w+5JzVTpRlf/B4BU8f/GWrnszCXIm+sOzn5JwDlqPDu61YsmA3kE9S
0rTPj/1o1rFdK8CuxpWA6lqM31zqQg==
-----END CERTIFICATE-----
`
wantFingerprint = "sha256:01b7ff2c19b3d8475e2228874aed3da996c66e8c0b32812998c39c68e0c59afb"
)

testDir := t.TempDir()
binDir := filepath.Join(testDir, "bin")
if err := os.Mkdir(binDir, 0o755); err != nil {
t.Fatalf("create fake binary directory: %v", err)
}
envPath := filepath.Join(testDir, "compute.yaml")
envYAML := `global:
image:
repository: example
agentConfig:
mergeConfig: |
agent:
logLevel: debug
cluster:
validationPolicy:
name: PreserveThisPolicy
workload:
stargateQUICInsecure: true
`
if err := os.WriteFile(envPath, []byte(envYAML), 0o600); err != nil {
t.Fatalf("write compute environment: %v", err)
}
// Feature setup updates unrelated dotted keys through the repository YAML
// editor, which reserializes the whole document before this script runs.
if err := dsl.UpdateYAMLKeys(envPath, [][2]string{{"global.image.repository", "feature-updated"}}); err != nil {
t.Fatalf("apply feature-style compute environment update: %v", err)
}
certificatePath := filepath.Join(testDir, "ca.pem")
if err := os.WriteFile(certificatePath, []byte(certificatePEM), 0o600); err != nil {
t.Fatalf("write test certificate: %v", err)
}

kubectlScript := `#!/usr/bin/env bash
set -euo pipefail
if [[ "$1" == "get" && "$2" == "secret" ]]; then
echo "root token must not be read for the public CA endpoint" >&2
exit 91
elif [[ "$1" == "port-forward" ]]; then
if [[ " $* " != *" --context k3d-test "* ]] || [[ " $* " != *" :8200 "* ]]; then
exit 2
fi
printf 'Forwarding from 127.0.0.1:43123 -> 8200\n'
exec /bin/sleep 300
else
exit 2
fi
`
curlScript := `#!/usr/bin/env bash
set -euo pipefail
if [[ " $* " == *" --config "* ]] || [[ " $* " == *"test-token"* ]]; then
exit 2
fi
if [[ " $* " != *" --connect-timeout "* ]] || [[ " $* " != *" --max-time "* ]]; then
exit 2
fi
if [[ " $* " != *" http://127.0.0.1:43123/"* ]]; then
exit 2
fi
python3 - "$FAKE_CA_FILE" <<'PYEOF'
import json
import pathlib
import sys

print(json.dumps({"data": {"certificate": pathlib.Path(sys.argv[1]).read_text()}}))
PYEOF
`
for name, body := range map[string]string{"kubectl": kubectlScript, "curl": curlScript} {
if err := os.WriteFile(filepath.Join(binDir, name), []byte(body), 0o755); err != nil {
t.Fatalf("write fake %s: %v", name, err)
}
}

cmd := exec.Command("bash", "scripts/write-transport-trust-env.sh", envPath, "k3d-test")
cmd.Env = append(os.Environ(), "FAKE_CA_FILE="+certificatePath, "PATH="+binDir+":"+os.Getenv("PATH"))
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("write transport trust environment: %v\n%s", err, out)
}

updated, err := os.ReadFile(envPath)
if err != nil {
t.Fatalf("read updated compute environment: %v", err)
}
var environment struct {
AgentConfig struct {
MergeConfig string `yaml:"mergeConfig"`
} `yaml:"agentConfig"`
}
if err := yaml.Unmarshal(updated, &environment); err != nil {
t.Fatalf("parse updated compute environment: %v", err)
}
var mergeConfig struct {
Agent struct {
LogLevel string `yaml:"logLevel"`
} `yaml:"agent"`
Cluster struct {
ValidationPolicy struct {
Name string `yaml:"name"`
} `yaml:"validationPolicy"`
} `yaml:"cluster"`
Workload struct {
StargateQUICInsecure bool `yaml:"stargateQUICInsecure"`
TransportTLS struct {
TrustMode string `yaml:"trustMode"`
TrustBundleFingerprint string `yaml:"trustBundleFingerprint"`
TrustBundlePEM string `yaml:"trustBundlePem"`
} `yaml:"transportTLS"`
} `yaml:"workload"`
}
if err := yaml.Unmarshal([]byte(environment.AgentConfig.MergeConfig), &mergeConfig); err != nil {
t.Fatalf("parse merged agent configuration: %v", err)
}
if got, want := mergeConfig.Agent.LogLevel, "debug"; got != want {
t.Fatalf("agent.logLevel = %q, want preserved value %q", got, want)
}
if got, want := mergeConfig.Cluster.ValidationPolicy.Name, "PreserveThisPolicy"; got != want {
t.Fatalf("cluster.validationPolicy.name = %q, want preserved value %q", got, want)
}
if mergeConfig.Workload.StargateQUICInsecure {
t.Fatal("workload.stargateQUICInsecure remained enabled with bundle trust")
}
if got, want := mergeConfig.Workload.TransportTLS.TrustMode, "bundle"; got != want {
t.Fatalf("workload.transportTLS.trustMode = %q, want %q", got, want)
}
if got := mergeConfig.Workload.TransportTLS.TrustBundleFingerprint; got != wantFingerprint {
t.Fatalf("workload.transportTLS.trustBundleFingerprint = %q, want %q", got, wantFingerprint)
}
if got := mergeConfig.Workload.TransportTLS.TrustBundlePEM; got != certificatePEM {
t.Fatalf("workload.transportTLS.trustBundlePem was not preserved exactly")
}
}

func TestWriteTransportTrustEnvDoesNotSendTokenWhenPortForwardFails(t *testing.T) {
testDir := t.TempDir()
binDir := filepath.Join(testDir, "bin")
if err := os.Mkdir(binDir, 0o755); err != nil {
t.Fatalf("create fake binary directory: %v", err)
}
envPath := filepath.Join(testDir, "compute.yaml")
if err := os.WriteFile(envPath, []byte("agentConfig:\n mergeConfig: |\n cluster: {}\n"), 0o600); err != nil {
t.Fatalf("write compute environment: %v", err)
}
curlLog := filepath.Join(testDir, "curl.log")
kubectlScript := `#!/usr/bin/env bash
set -euo pipefail
if [[ "$1" == "get" && "$2" == "secret" ]]; then
echo "root token must not be read for the public CA endpoint" >&2
exit 91
elif [[ "$1" == "port-forward" ]]; then
printf 'unable to listen on any requested port\n' >&2
exit 42
else
exit 2
fi
`
curlScript := `#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' "$*" >>"$FAKE_CURL_LOG"
exit 7
`
sleepScript := `#!/usr/bin/env bash
exit 0
`
for name, body := range map[string]string{
"kubectl": kubectlScript,
"curl": curlScript,
"sleep": sleepScript,
} {
if err := os.WriteFile(filepath.Join(binDir, name), []byte(body), 0o755); err != nil {
t.Fatalf("write fake %s: %v", name, err)
}
}

cmd := exec.Command("bash", "scripts/write-transport-trust-env.sh", envPath, "k3d-test")
cmd.Env = append(os.Environ(), "FAKE_CURL_LOG="+curlLog, "PATH="+binDir+":"+os.Getenv("PATH"))
out, err := cmd.CombinedOutput()
if err == nil {
t.Fatalf("write transport trust environment unexpectedly succeeded\n%s", out)
}
if !strings.Contains(string(out), "port-forward exited before becoming ready") {
t.Fatalf("failure did not identify the port-forward readiness error:\n%s", out)
}
if body, readErr := os.ReadFile(curlLog); readErr == nil && len(body) > 0 {
t.Fatalf("curl was invoked after port-forward failure: %s", body)
} else if readErr != nil && !os.IsNotExist(readErr) {
t.Fatalf("read curl log: %v", readErr)
}
}

func TestResolveGatewayDomainUsesResolvedIPv4(t *testing.T) {
cmd := exec.Command("bash", "scripts/resolve-gateway-domain.sh", "gateway.example.invalid")
cmd.Env = append(os.Environ(), "EKS_GATEWAY_IPV4=192.0.2.10")
Expand Down
17 changes: 12 additions & 5 deletions tests/bdd/godog_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,13 @@ controlPlane:
reval: reval.localhost
nats: nats.localhost
invocation: invocation.localhost
managementTls:
trustMode: bundle
caBundlePem: test-ca-bundle
transportTls:
trustMode: bundle
trustBundleFingerprint: sha256:test-fingerprint
trustBundlePem: test-ca-bundle
`
writeArtifact(t, repoRoot, "self-managed", "control-plane-profile.yaml", body)
}
Expand Down Expand Up @@ -511,7 +518,7 @@ func TestSingleClusterHelmfileFeatureFileWiresToSteps(t *testing.T) {

// TestSingleClusterHelmfileLLMPKIFeatureFileWiresToSteps runs the
// LLM PKI Helmfile feature against a fake runner, with canned results
// for the trust script, the LLM invoke, and the no-auth curl.
// for the LLM invoke and the no-auth curl.
func TestSingleClusterHelmfileLLMPKIFeatureFileWiresToSteps(t *testing.T) {
t.Setenv("NGC_API_KEY", "test-key")
t.Setenv("SAMPLE_NGC_ORG", "test-org")
Expand All @@ -520,9 +527,9 @@ func TestSingleClusterHelmfileLLMPKIFeatureFileWiresToSteps(t *testing.T) {
t.Setenv("REPO_ROOT", "/repo-root-placeholder")
suite := newWiringSuite(t, newFakeRunner(map[string]harness.Result{
"helm list --all-namespaces --kube-context k3d-ncp-local -o json": {ExitCode: 0, Stdout: helmListAllNamespacesJSON()},
"tests/bdd/scripts/write-transport-trust-env.sh deploy/stacks/nvcf-compute-plane/environments/local-bdd-pki.yaml k3d-ncp-local": {
"helm get values nvca-operator --namespace nvca-operator --kube-context k3d-ncp-local -o yaml": {
ExitCode: 0,
Stdout: "wrote transportTLS bundle config (fingerprint sha256:abc) to deploy/stacks/nvcf-compute-plane/environments/local-bdd-pki.yaml\n",
Stdout: "agentConfig:\n mergeConfig: |\n workload:\n stargateQUICInsecure: false\n transportTLS:\n trustMode: bundle\n trustBundleFingerprint: sha256:test\n",
},
"/usr/bin/nvcf-cli --config /repo-root-placeholder/tests/bdd/fixtures/nvcf-cli-local.yaml function invoke" +
" --inference-url /v1/chat/completions --model-name openai-compatible-sample" +
Expand Down Expand Up @@ -1490,7 +1497,7 @@ agentConfig:
validationPolicy:
name: Unrestricted
workload:
stargateQUICInsecure: true
stargateQUICInsecure: false
`)
}

Expand All @@ -1510,7 +1517,7 @@ agentConfig:
validationPolicy:
name: Unrestricted
workload:
stargateQUICInsecure: true
stargateQUICInsecure: false
`)
}

Expand Down
Loading