test(bdd): single-cluster PKI feature with a secured LLM invoke

[along-2017]

Parent

Relates to #1019.

What was built

PR #1075 adds a single-cluster PKI feature with its own live entry point and
wiring test. It reuses the existing fixtures and overlays the LLM PKI add-on
and secure stargate QUIC transport without changing the default fixtures or
existing feature behavior.

The merged design uses the canonical control-plane profile as the trust
handoff. The exported profile carries non-empty management and transport bundle
trust; compute registration and the NVCA install consume and validate that
trust without reading or sending an OpenBao root token.

The live scenarios cover generated PKI render, control-plane install, issuer
and router certificate readiness, profile export, compute registration, secure
NVCA trust settings, backend health, LLM function deployment, authenticated
routed invocation, expected unauthenticated access rejection, and deployment
cleanup.

Acceptance criteria

• PKI release, issuer, and router certificate asserted Ready
• Canonical profile bundle trust distributed without root-token access or
  secrets in command logs
• LLM chat completion and 401 checks pass over the secured tunnel
• Existing features and fixtures unchanged
• Wiring test, short suite, automatic checks, and fresh live run green

Completion evidence

PR #1075 exact head dcb58f71377fbda571098a810d66f1cb77a2799f
merged as fc10e13b155399424bc46fe0380b34644e1d8350 after approval and green
automatic checks.

The final fresh bounded run passed 4/4 scenarios and 69/69 steps in 14m34s.
The fixed-response workload verifies routing and contract wiring; it does not
measure token-generation performance.

Publication of the validated compatible standalone-stack input remains a
separate release-delivery follow-up and does not change this repository's
completed BDD implementation.

[mikeyrcamp]

Current implementation: #1075 at 702e37fa.

Repository verification at that head:

• go test -run '^TestSingleClusterHelmfileLLMPKIFeatureFileWiresToSteps$' -count=1 — PASS
• go test -short -count=1 ./... — PASS for nvcf-bdd, dsl, harness, and steps
• go vet ./... — PASS
• bash -n scripts/write-transport-trust-env.sh — PASS
• git diff --check — PASS

Live result: STILL BLOCKED before invocation/cleanup.

The documented live feature ran at 18c601de with Helmfile 1.1.9 and Helm
3.15.4. PKI render/install, issuer readiness, stargate certificate readiness,
trust-bundle fingerprint generation, NVCA rollout, and backend health all
passed. Function creation then passed, but deployment returned HTTP 400:

Invalid InstanceType 'NCP.GPU.H100_8x' specified

Godog stopped as required: 4 scenarios (3 passed, 1 failed), 60 steps
(48 passed, 1 failed, 11 skipped). The LLM invocation and deployment cleanup
were not run after the failure.

This is a feature-fixture configuration defect rather than observed PKI product
failure. Current local k3d BDD scenarios and the CLI default use
NCP.GPU.H100_1x; H100_8x is used by the separate EKS scenario. Commit
702e37fa changes this feature to H100_1x and adds a focused assertion for
the exact deployment command. The corrected branch has repository verification
but has not had a fresh live run yet.

Precise next action: run
go test -run '^TestSingleClusterHelmfileLLMPKI$' -timeout 90m -v on a fresh
documented local stack with the repository-pinned toolchain, and verify the
secured invocation plus cleanup at 702e37fa or its successor.

[mikeyrcamp]

RESOLVED — corrected-head live validation passed.

PR #1075 commit 702e37faef805718276e20029545caac34d4aeb5 was validated from a new isolated worktree using:

PATH="<standalone-stack-bin>:$PATH" go test -run '^TestSingleClusterHelmfileLLMPKI$' -timeout 90m -v

Toolchain: Go 1.26.4, k3d 5.8.3, kubectl 1.36.2, Helm 3.15.4, and Helmfile 1.1.9.

Direct evidence from the run:

• PKI manifest rendering and the self-managed control-plane install passed.
• ClusterIssuer/nvcf-openbao-pki and Certificate/nvcf/stargate-quic-tls reached Ready.
• The generated transport-trust helper emitted the canonical fingerprint sha256: marker.
• NVCA installed successfully and NVCFBackend/ncp-local reported agentStatus=healthy.
• Function creation and deployment using NCP.GPU.H100_1x passed.
• API-key generation passed.
• The secured routed invocation returned chat.completion and the expected fixed 128-byte response.
• The unauthenticated gateway request returned HTTP 401.
• function delete --deployment-only passed, verifying deployment cleanup.
• Final Godog result: 4 scenarios passed, 60 steps passed, 5m50.449320633s. Go test completed successfully in 369.650s.

Read-only post-run checks still showed the issuer and certificate Ready, the NVCA deployment 1/1 available, and the backend healthy.

This proves the single-cluster PKI/trust/routing contract for the fixed-response sample. It is not token-generation capacity or latency evidence.

[mikeyrcamp]

Follow-up repository verification for PR #1075 head 5f18ab15:

• Split the 296/302-character canned curl recorder key into adjacent raw string
  literals, preserving the exact runtime command while satisfying the 250-line
  source limit.
• go test -run '^TestSingleClusterHelmfileLLMPKIFeatureFileWiresToSteps$' -count=1 — PASS
• go test -short -count=1 ./... — PASS (nvcf-bdd, dsl, harness, and steps)
• go vet ./..., helper bash syntax, source line-length, and git diff --check — PASS
• CodeRabbit finding resolved after validation.

The documented local golangci-lint command remains host-toolchain blocked:
golangci-lint 2.11.4 was built with Go 1.26.2, while the host uses Go 1.27;
go list ./... successfully enumerates all four BDD packages.

No live rerun was performed for this source-layout-only change. The successful
60/60-step live result at behavior-identical parent 702e37fa remains the
direct live evidence.

[mikeyrcamp]

Fresh current-head validation: STOPPED / tokenless path NOT TESTED.

PR #1075 head 4f6ab883d943a85d007cddcdea804188135d08d5 remains repository-verified, but a fresh live run stopped at the first unexpected repository-documented cleanup failure, before applying a validation-only tokenless change, building the CLI, creating a cluster, or running the BDD feature:

$ make -C tools/ncp-local-cluster destroy-all-ncp-local
/bin/sh: 1: set: Illegal option -o pipefail
make: *** [Makefile:156: destroy-all-ncp-local] Error 2

GNU Make 4.3 selected /bin/sh (dash 0.5.12), while tools/ncp-local-cluster/Makefile:159 uses Bash-only set -o pipefail without selecting Bash. Public main 603059d81a3adc4f134866dbb0f678b053e59e0c contains the same recipe; PR #1075 does not introduce it. #1261 tracks the portable-cleanup fix and focused acceptance criteria.

Final source review also found that ReadPKICertificatePEM still reads the OpenBao root-token Secret and places X-Vault-Token in transient kubectl run container arguments for the public /cert/ca response. This behavior predates #1075, but its profile-export scenario exercises it. The next live attempt should remove that Secret lookup/header, add CLI-level regression coverage proving neither occurs, fix #1261, and run the complete feature at the exact resulting head.

The prior 4/4-scenario, 60/60-step pass at 702e37fa remains valid prior live evidence; it is not direct evidence for current-head or tokenless CA retrieval.

[mikeyrcamp]

Fresh current-head validation: second stop identified a PR-local step-context defect.

Using the previously recorded scoped Bash workaround for #1261, a fresh run of PR #1075 head 4f6ab883d943a85d007cddcdea804188135d08d5 entered the feature but stopped after six setup/file steps. The first raw command, k3d cluster get ncp-local-cp, never executed: its command log had zero-byte stdout/stderr and the runner returned context canceled. No cluster was created, and ledger restoration removed all generated inputs.

Direct source evidence shows the cause:

• the live Before hook returns BeginSignalSafeStep's child context;
• normal After calls EndSignalSafeStep, which cancels that child; and
• After returns the same canceled context, which Godog propagates to the following step.

File-only steps do not observe the canceled context; the first CommandRunner does, making live commands structurally unable to start after an earlier step.

Focused TDD on an isolated, unpushed validation patch captured RED:

go test ./harness -run '^TestSignalSafeStepsPreserveNormalContinuation$' -count=1
second step inherited canceled context: context canceled

The minimal local fix retains and returns the uncanceled parent context from normal step completion while preserving the existing once-only signal cleanup. It also preserves context values and repeated-end behavior. Verification on the combined local tokenless/context patch:

go test -race ./harness -count=1                         # PASS
MAKEFLAGS='SHELL=/bin/bash' go test -short ./... -count=1 # PASS
go vet ./...                                              # PASS
go test ./internal/openbao -count=1                       # PASS
git diff --check                                          # PASS

The PR remote head remains unchanged and still contains both merge blockers: the canceled-context propagation defect and the root-token-bearing public-CA read. A fresh live retry has not yet occurred.