Skip to content

local-development.md

Latest commit

 

History

History
132 lines (103 loc) · 4.81 KB

local-development.md

File metadata and controls

132 lines (103 loc) · 4.81 KB
Raw

Local Development (k3d)

Run a full NVCF self-hosted stack on your laptop using k3d for development, testing, or demos. The canonical local-k3d tooling lives at tools/ncp-local-cluster/ in this repo.

This setup is only for local development. It uses fake GPUs, a single Cassandra replica, and ephemeral storage. Do not use this for production workloads.

Clone the public repository before using any local development flow:

git clone https://github.com/nvidia/nvcf.git
cd nvcf

Pick a flow

Four canonical flows are covered below: pick a topology from Topologies and an install path from Install paths.

Topology Install path
Single-cluster CLI (nvcf-cli self-hosted install)
Single-cluster Helmfile (make install HELMFILE_ENV=...)
Multi-cluster CLI (nvcf-cli self-hosted install)
Multi-cluster Helmfile (make install HELMFILE_ENV=...)

Topologies

Single-cluster topology: This brings up one k3d cluster named ncp-local. Control plane and compute plane share the cluster. The fastest path for function-lifecycle testing and basic install validation.

Multi-cluster topology: This brings up ncp-local-cp plus ncp-local-compute-N (N=1 by default). Control plane lives on the cp cluster; compute plane on the compute cluster. Required when you need to exercise cross-cluster registration, the OIDC/JWKS discovery flow, or the .test hostname routing the cp Gateway exposes to compute workers.

The two topologies cannot run at the same time. Both claim host ports 8080/8443/4222. Destroy one before bringing up the other:

# from single-cluster -> multi-cluster
make -C tools/ncp-local-cluster destroy

# from multi-cluster -> single-cluster
make -C tools/ncp-local-cluster destroy-multicluster

Install paths

CLI path: The profile-and-values workflow installs the control plane, writes a control-plane profile, registers the compute plane, and installs the generated compute-plane values. For a repository-built CLI, run these commands from the repository root:

nvcf-cli self-hosted \
  --control-plane-stack deploy/stacks/self-managed \
  install --control-plane

nvcf-cli init

nvcf-cli self-hosted \
  --compute-plane-stack deploy/stacks/nvcf-compute-plane \
  compute-plane register \
  --control-plane-profile deploy/stacks/self-managed/out/control-plane-profile.yaml \
  --cluster-name <cluster-name> \
  --kube-context <compute-plane-context>

nvcf-cli self-hosted \
  --compute-plane-stack deploy/stacks/nvcf-compute-plane \
  compute-plane install \
  --values deploy/stacks/nvcf-compute-plane/out/<cluster-name>-register-values.yaml \
  --kube-context <compute-plane-context>

The alternative self-hosted install --compute-plane command registers the cluster itself before applying the compute-plane manifests. Do not run cluster register before that combined command. Use one compute-plane workflow or the other.

Repository-built CLI binaries do not contain the stack OCI defaults injected into packaged releases. The commands above pass the required stack explicitly. Use both stack directories from the same checkout. The CLI selects in-cluster or cross-cluster URLs from the kube contexts you pass.

Helmfile path: Helmfile drives the install through split Make targets: deploy/stacks/self-managed/Makefile for control plane (make template, make install) and deploy/stacks/nvcf-compute-plane/Makefile for compute plane (make register-cluster, make install). The operator authors topology-correct URLs into an environment file (different fixture per topology) instead of relying on a CLI-managed profile.

The two install paths intentionally diverge. See tests/bdd/AGENTS.md (the "CLI vs Helmfile install paths" section) for the rationale.

Prerequisites (common to all flows)

  • Docker (running)

  • k3d v5.x or later

  • kubectl

  • helm >= 3.12

  • helmfile >= 1.1.0, < 1.2.0 (Helmfile flows only)

  • helm-diff plugin (Helmfile flows only): helm plugin install https://github.com/databus23/helm-diff

  • An NGC API key with access to the NVCF chart and image registry.

  • nvcf-cli built from this repo:

    go build -C src/clis/nvcf-cli -o ../../../nvcf-cli .

Cleanup

The BDD suite ships destructive cleanup helpers reused for hand-driven local dev:

Scope Command
Stack-only (helm releases) on single-cluster tests/bdd/scripts/destroy-stack.sh single
Stack-only on multi-cluster tests/bdd/scripts/destroy-stack.sh multi
Whole single-cluster topology make -C tools/ncp-local-cluster destroy
Whole multi-cluster topology make -C tools/ncp-local-cluster destroy-multicluster
Every ncp-local* k3d cluster on the host make -C tools/ncp-local-cluster destroy-all-ncp-local