Skip to content

Track runtime security updates through image, chart, and stack releases #2230

Description

@sbaum1994

Track the runtime security updates from dependency changes through published service images, Helm charts, stack pins, and the later 1.0.x backport.

The first dependency change updates the shared Go SDK from 1.26.5 to 1.26.8 and the Rust distroless cc runtime from v4.1.2 to v4.1.4. The candidate runtime was inspected on amd64 and arm64 and contains libc6 2.41-12+deb13u4. OpenBao migration helper dependencies are being checked separately so existing source fixes are retained.

The delivery order is:

  1. Merge reviewed dependency fixes and record their merge commits.
  2. Publish all affected producer releases and verify their image digests and embedded versions.
  3. Update every consuming Helm chart, including indirect worker and hook image references, and publish the charts.
  4. Consume the verified chart versions in the control-plane and compute-plane stacks; validate their resolved image inventories.
  5. Repeat the sequence for the 1.0.x maintenance release using the appropriate service and chart maintenance trains.

The Go SDK consumers in this batch are ess-agent, llm-api-gateway, nvca (both agent and operator images), vanity-gateway, grpc-proxy, image-credential-helper, nats-auth-callout, ratelimiter, worker-init, worker-llm-credentials, worker-utils, and helm-reval. The Rust runtime consumers are function-autoscaler, http-invocation, and stargate.

Release tracking requirements:

  • Keep one row per image with dependency PR/merge commit, producer tag, published image digest, chart/image-value path, chart PR/version, stack PR/version, replacement scan, and backport status.
  • A shared root SDK/base edit does not itself trigger every service's path-scoped semantic release. Record and complete the producer release work before consuming new tags in charts.
  • Do not guess future image or chart versions. Select immutable published artifacts with verified contents.
  • Preserve separate commits for the toolchain and runtime updates so a maintenance backport can select the relevant fix without unrelated feature changes.
  • For the 1.0.x stack, establish the service/chart source revisions from the released inventory. A cherry-pick only to the stack branch does not rebuild its dependencies.

Existing downstream PRs to reconcile after the new artifacts are available:

Completion requires verified replacement artifacts and resolved stack inventories, followed by recorded backport evidence. The dependency PR alone does not complete this issue.

Dependency PRs opened 2026-10-02

Both PRs target main. Publication, chart consumption, stack consumption, replacement scans, and 1.0.x backports remain pending. Record the final merge SHA as well as the review commits.

Image Dependency PR 1.0 inventory image version Image release Chart update/release Stack update/release Backport
ess-agent #2231 1.4.1 Pending Pending Pending Pending
llm-api-gateway #2231 0.14.2 Pending Pending Pending Pending
nvca-operator #2231 3.10.0 Pending Pending Pending Pending
nvca #2231 3.10.0 Pending Pending Pending Pending
nvcf-ai-api-gateway-service #2231 1.35.1 Pending Pending Pending Pending
nvcf-cassandra-migrations Cassandra follow-up 0.17.6 Pending Pending Pending Pending
nvcf-function-autoscaler #2231 1.21.8 Pending Pending Pending Pending
nvcf-grpc-proxy #2231 1.33.5 Pending Pending Pending Pending
nvcf-image-credential-helper #2231 0.11.1 Pending Pending Pending Pending
nvcf-invocation-service #2231 0.12.1 Pending Pending Pending Pending
nvcf-nats-auth-callout-service #2231 0.8.3 Pending Pending Pending Pending
nvcf-openbao-migrations #2232 0.19.5 Pending Pending Pending Pending
nvcf-ratelimiter #2231 1.17.3 Pending Pending Pending Pending
nvcf-worker-init-oss #2231 1.2.1 Pending Pending Pending Pending
nvcf-worker-llm-credentials-oss #2231 1.1.2 Pending Pending Pending Pending
nvcf-worker-utils-oss #2231 1.2.3 Pending Pending Pending Pending
reval-server #2231 0.20.2 Pending Pending Pending Pending
stargate #2231 0.18.0 Pending Pending Pending Pending

Backport baselines: self-managed 1.0.1 inventory and compute-plane 1.0.0 inventory. The producer source tags and commits for all 18 image versions have been resolved and recorded in the working ledger.

Additional consumer paths requiring explicit review:

  • deploy/helm/cloud-functions/nvcf-api/values.yaml: worker and ESS image strings under api.remoteConfig.configData.nvcf.sidecars.
  • deploy/helm/cloud-tasks/nvct-api/values.yaml: separate ESS image string under nvctApi.remoteConfig.configData.nvct.sidecars.
  • deploy/helm/nvca-operator/nvca-operator/values.yaml: both credential-helper modes, plus operator/agent version defaults and compute-plane overrides.
  • deploy/helm/llm-request-router/llm-request-router/values.yaml: Stargate image and backend-router fallback.
  • deploy/stacks/self-managed/global.yaml.gotmpl: OpenBao migration image fallback for PKI, the UI hook's minimum version, and independent addons.lls.hmacRotation.image.tag for SIS/LLS. Render all optional profiles so the new OpenBao chart does not leave these hooks on an older image.

The chart and stack versions will be filled in only after the corresponding artifacts have been published and verified. Existing accumulated bump PRs must be filtered to the intended changes for maintenance backports.

Helm decision updated 2026-10-02: the official Helm 3.22.0 binary was evaluated, but govulncheck reports additional crypto advisories. The final PR removes the unused dependency and all proposed Helm-specific build/verification pins. Both verifier review comments therefore concern code no longer in the PR. Live BDD validation found that OpenBao chart 0.32.6 unconditionally checks Helm inside its initialization Job. PR #2232 now also gates that check to standalone script mode, with regression coverage (commit f5f601c2a6d113242fed2e2fb8200ab0e82bc39b). The net backport must include the package removal, final-image absence check, chart prerequisite fix, and compatibility documentation. The Helm-free image cannot be consumed by the old chart alone. Publication and replacement scans remain pending.

Local validation completed at f5f601c2a: single-cluster Helmfile BDD passed 10 scenarios/89 steps; multi-cluster passed 10 scenarios/93 steps. Both used the Helm-free image and corrected OpenBao chart. Initialization, core migrations, and LLM PKI hooks completed with the exact built image; live container checks confirmed Helm absence. Multi-cluster required the supported HTTP callback fixture override tracked in #2245. UI and SIS/LLS were disabled and need separate release QA. No published image, chart, stack release, or backport is claimed by these tests.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions