feat(#66): test environment layer — provisioning, config discovery, test data, config apply

.gitignore

@@ -17,6 +17,9 @@ log
 # Research output files
 outputs/
 
+# Managed cht-core checkout (scripts/test-env-up.sh with no path)
+.cht-core/
+
 # Pipeline skip-audit log. Every real run appends to it, so tracking it made each
 # run produce a commit and let a stray test entry reach real pipeline data (#146).
 agent-memory/_skipped.ndjson

designs/layer_recommendations/test-environment-layer.md

@@ -223,7 +223,7 @@ Default: **CouchDB wipe + reseed** for config/data iterations. Under Model A, **
 
 #### Teardown
 
-`docker compose down -v` (the `-v` clears volumes for a clean slate). Always run on completion or failure to avoid orphaned containers.
+`scripts/test-env-down.sh` (`docker compose down -v`). `-v` removes the named volumes, but CouchDB data is a bind mount under `local-build/` and survives, so a clean database also means deleting that directory. Always run on completion or failure to avoid orphaned containers.
 
 ### 2. Config Discovery
 

docker/cht-agent-net.override.yml

@@ -0,0 +1,39 @@
+# cht-agent-net.override.yml
+#
+# Layers onto cht-core's local-build compose (or the docker-helper stack) so the
+# running CHT instance joins the shared `cht-agent-net` network that the
+# dockerized cht-agent is also attached to. This lets the agent reach the
+# instance at https://nginx with no host-port juggling and — importantly — with
+# NO Docker access (the agent only polls /api/v2/monitoring).
+#
+# HUMAN-run only (the cht-agent never runs Docker). scripts/test-env-up.sh applies
+# it with a per-checkout project, internal network and data dir (scripts/lib/test-env.sh);
+# by hand, pass the same values that test_env_compose passes, not only -p. Without
+# CHT_NETWORK, the stack shares the default `cht-net` network and the couchdb/haproxy/api
+# names with any other CHT stack that also uses `cht-net`. Without COUCHDB_PASSWORD,
+# Compose stops. Use this command:
+#   COUCHDB_USER=medic COUCHDB_PASSWORD=password COMMON_NAME=nginx \
+#   NGINX_HTTP_PORT=127.0.0.1:80 NGINX_HTTPS_PORT=127.0.0.1:443 \
+#   CHT_NETWORK=<project>-net COUCHDB_DATA=./srv-<project> \
+#     docker compose -p <project> -f cht-couchdb.yml -f cht-core.yml -f cht-agent-net.override.yml up -d
+# Only one stack's nginx can be on cht-agent-net at a time: `nginx` would resolve to both.
+#
+# VALIDATE before relying on this: the service name (`nginx`) and the key of the
+# stack's internal network (`cht-net` below; CHT_NETWORK sets its actual name) must
+# match what `npm run local-images` generates in `local-build/` for your cht-core
+# version. Confirm with:
+#   docker compose ... config            # see resolved service/network names
+#   docker network inspect cht-agent-net # confirm nginx is attached
+# If the internal network differs, update the first entry under nginx.networks.
+
+networks:
+  cht-agent-net:
+    external: true
+
+services:
+  # nginx is the HTTPS entry point the agent reaches. Keep it on the stack's
+  # internal network AND join the shared cht-agent-net.
+  nginx:
+    networks:
+      - cht-net
+      - cht-agent-net

docker/docker-compose.cht-agent.yml

@@ -48,6 +48,8 @@ services:
       # .credentials.json mount below (for OAuth-based tools).
       ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-}
       CHT_CORE_PATH: /workspace/cht-core
+      # The same working copy on the host, where the operator runs the printed gates.
+      CHT_CORE_HOST_PATH: ${CHT_CORE_PATH}
       # CHT entry point on cht-agent-net (nginx joins via the #66 override)
       CHT_URL: ${CHT_URL:-https://nginx}
       NODE_ENV: development

docs/docker-agent-runbook.md

@@ -129,17 +129,41 @@ When the Test Environment Layer (#66) requests an environment, the agent
 *waits and polls* — you bring it up:
 
 ```bash
-# Build images from the agent's edited working copy (Model A rebuild-on-change)
-cd ~/src/cht-core && npm run local-images
-
-# Start the stack, attaching nginx to cht-agent-net via the #66 override
-docker compose -f <cht-core compose files> \
-  -f <cht-agent repo>/docker/cht-agent-net.override.yml up -d
+# From the cht-agent repo: build the agent's edited working copy into images
+# (Model A rebuild-on-change) and start it, nginx joined to cht-agent-net
+CHT_CORE_REBUILD=1 scripts/test-env-up.sh ~/src/cht-core
 ```
 
-> The override file `docker/cht-agent-net.override.yml` ships with #66.
-> Validate service/network names against your generated compose before
-> relying on it (see the comments in that file).
+> `scripts/test-env-up.sh` gives each checkout its own Compose project, internal
+> network and CouchDB data dir, and refuses to start while another stack's nginx
+> is on `cht-agent-net`. Overrides are listed in `scripts/lib/test-env.sh`.
+>
+> The stack's cert is self-signed, and the agent's readiness poll stops at the first
+> TLS verification error. So bring the stack up before the agent run starts, not
+> after the agent asks: the poll then succeeds at once. Run the `docker cp` line
+> that `test-env-up.sh` prints at its end, then copy that file into the container
+> (`docker cp <file> cht-agent:/tmp/cht-cert.pem`). Start the run with
+> `docker exec -e NODE_EXTRA_CA_CERTS=/tmp/cht-cert.pem ...`. After
+> `test-env-down.sh`, the next bring-up makes a new cert. If the bring-up used
+> non-default `COUCHDB_USER` or `COUCHDB_PASSWORD`, pass the same values with `docker exec -e`.
+
+Settings the layer reads from the environment:
+
+| Variable | Read by | Effect (default) |
+|---|---|---|
+| `CHT_URL` | agent | Instance `provision` dials when the call passes no `url` (`https://nginx`). |
+| `COUCHDB_USER` / `COUCHDB_PASSWORD` | agent, scripts | Stack admin; each defaults on its own (`medic` / `password`). The agent refuses the default password against a host it does not treat as disposable. |
+| `CHT_TEST_ENV_ALLOW_EXTERNAL=1` | agent | Allows a host it does not treat as disposable, like `allowExternalTarget`. |
+| `CHT_CONF_BIN` | agent | cht-conf binary to spawn (`cht`). |
+| `CHT_CORE_HOST_PATH` | agent | Host path of the working copy, printed in the gates. The agent's compose file sets it to `$CHT_CORE_PATH`. |
+| `CHT_TEST_ENV_PROJECT` | scripts, agent | Compose project (`cht-agent-<dir>-<path hash>`). The agent repeats it in the gates it prints. |
+| `CHT_CORE_PATH` | scripts | Working copy, when no path argument is given. |
+| `CHT_CORE_CLONE_DIR` | scripts | Managed checkout when neither is given (`<repo>/.cht-core`), cloned from `CHT_CORE_UPSTREAM` at `CHT_CORE_BRANCH` (medic/cht-core, master). |
+| `CHT_CORE_REBUILD=1` | scripts | Runs `npm run build-dev` even when a build exists. |
+| `COUCHDB_DATA` | scripts | CouchDB bind mount (`local-build/srv-<project>`). It survives `test-env-down.sh`. |
+| `NGINX_HTTP_PORT` / `NGINX_HTTPS_PORT` | scripts | Host binds (`127.0.0.1:80` / `127.0.0.1:443`). Opening them to the LAN needs a non-default `COUCHDB_PASSWORD`. |
+| `COMMON_NAME` | scripts | Cert CN (`nginx`, the host the agent dials). |
+| `NODE_EXTRA_CA_CERTS` | Node | The stack's `cert.pem`, read once per process (see above). |
 
 The agent detects readiness via `GET https://nginx/api/v2/monitoring` and
 continues. CouchDB-tier resets it does itself over HTTP; container restarts
@@ -161,7 +185,7 @@ git push origin cht-agent/<ticket>
 
 ```bash
 docker compose -f docker/docker-compose.cht-agent.yml down
-# CHT stack: docker compose -f <cht-core compose files> down -v
+scripts/test-env-down.sh ~/src/cht-core   # CHT stack
 # Full cleanup of the shared network (only once nothing else uses it):
 docker network rm cht-agent-net
 ```
@@ -173,5 +197,5 @@ docker network rm cht-agent-net
 | compose fails mounting `git-config.hardened` over `.git/config` | working copy is a git worktree (`.git` is a file) or missing. Run `docker/scripts/bootstrap-workspace.sh` on the host. |
 | `Sandbox verification FAILED` in `docker logs` | a hardening layer is missing — read which `[FAIL]` fired; don't bypass it. |
 | working-copy files unwritable from container | host uid ≠ 1000. Rebuild with a matching uid or chown the copy. |
-| agent can't reach `https://nginx` | CHT stack not attached to `cht-agent-net` — re-run compose with the #66 override; `docker network inspect cht-agent-net` should list nginx. |
+| agent can't reach `https://nginx` | CHT stack not attached to `cht-agent-net` — bring it up with `scripts/test-env-up.sh`; `docker network inspect cht-agent-net` should list one nginx. |
 | `fetch` fails for an `ssh://` or `git@` URL | by design (no ssh binary). Use the `https://` URL. |

scripts/lib/test-env.sh

@@ -0,0 +1,91 @@
+# shellcheck shell=bash
+# Shared by scripts/test-env-{up,down,restart}.sh (sourced, not run).
+#
+# Every cht-core checkout builds in a directory named `local-build`, and Compose
+# names a project after its directory, so without -p all checkouts share one project
+# and `down -v` on one deletes another's stack. Each checkout therefore gets its own
+# project, and with it its own internal network (the couchdb/api/haproxy names),
+# CouchDB bind mount, and cert volume. Only nginx joins the shared cht-agent-net.
+#
+# Env overrides:
+#   CHT_TEST_ENV_PROJECT  Compose project (default cht-agent-<dir>-<path hash>)
+#   COUCHDB_USER / COUCHDB_PASSWORD  admin (default medic / password, the agent's DEFAULT_AUTH)
+#   COUCHDB_DATA          CouchDB bind mount (default local-build/srv-<project>)
+#   NGINX_HTTP_PORT / NGINX_HTTPS_PORT  host binds (default 127.0.0.1:80 / 127.0.0.1:443;
+#                         set 443 to reach the stack from another device, and then also set
+#                         COUCHDB_PASSWORD: the default admin is medic / password)
+#   COMMON_NAME           cert CN (default nginx, the host name the containerized agent uses)
+
+TEST_ENV_REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+TEST_ENV_OVERRIDE="$TEST_ENV_REPO_ROOT/docker/cht-agent-net.override.yml"
+# Must match the external network in docker/cht-agent-net.override.yml.
+# shellcheck disable=SC2034 # used by test-env-up.sh
+TEST_ENV_NETWORK="cht-agent-net"
+
+test_env_default_target() {
+  local path="${1:-}"
+  printf '%s\n' "${path:-${CHT_CORE_PATH:-${CHT_CORE_CLONE_DIR:-$TEST_ENV_REPO_ROOT/.cht-core}}}"
+  return $?
+}
+
+test_env_hash() {
+  if command -v sha256sum >/dev/null 2>&1; then sha256sum; else shasum -a 256; fi | cut -c1-8
+  return $?
+}
+
+# Sets TEST_ENV_TARGET (physical path) and TEST_ENV_PROJECT for an existing checkout.
+test_env_select() {
+  local path="$1" base
+  TEST_ENV_TARGET="$(cd "$path" && pwd -P)"
+  base="$(basename "$TEST_ENV_TARGET" | LC_ALL=C tr '[:upper:]' '[:lower:]' | LC_ALL=C sed -e 's/[^a-z0-9_-]/-/g' -e 's/^-*//')"
+  TEST_ENV_PROJECT="${CHT_TEST_ENV_PROJECT:-cht-agent-${base}-$(printf '%s' "$TEST_ENV_TARGET" | test_env_hash)}"
+  return $?
+}
+
+test_env_require_build() {
+  local path="$1"
+  if [[ ! -d "$path/local-build" ]]; then
+    echo "error: no cht-core build at $path (pass a path or set CHT_CORE_PATH)" >&2
+    exit 1
+  fi
+}
+
+# COUCHDB_PASSWORD goes to every subcommand: the compose files declare ${COUCHDB_PASSWORD:?...}.
+test_env_compose() {
+  ( cd "$TEST_ENV_TARGET/local-build" &&
+    COUCHDB_USER="${COUCHDB_USER:-medic}" COUCHDB_PASSWORD="${COUCHDB_PASSWORD:-password}" \
+    CHT_NETWORK="$TEST_ENV_PROJECT-net" COUCHDB_DATA="${COUCHDB_DATA:-./srv-$TEST_ENV_PROJECT}" \
+    NGINX_HTTP_PORT="${NGINX_HTTP_PORT:-127.0.0.1:80}" NGINX_HTTPS_PORT="${NGINX_HTTPS_PORT:-127.0.0.1:443}" \
+    COMMON_NAME="${COMMON_NAME:-nginx}" \
+    docker compose -p "$TEST_ENV_PROJECT" -f cht-couchdb.yml -f cht-core.yml -f "$TEST_ENV_OVERRIDE" "$@" )
+  return $?
+}
+
+# Refuse a project that another checkout started. With a shared CHT_TEST_ENV_PROJECT,
+# up would recreate that checkout's containers, and down -v would delete them and its volumes.
+test_env_require_owner() {
+  local dirs dir
+  dirs="$(docker ps -a --filter "label=com.docker.compose.project=$TEST_ENV_PROJECT" \
+    --format '{{.Label "com.docker.compose.project.working_dir"}}' | sort -u)"
+  while IFS= read -r dir; do
+    if [[ -n "$dir" && "$dir" != "$TEST_ENV_TARGET/local-build" ]]; then
+      echo "error: Compose project '$TEST_ENV_PROJECT' has containers from $dir, which is not $TEST_ENV_TARGET/local-build." >&2
+      exit 1
+    fi
+  done <<< "$dirs"
+  return 0
+}
+
+# restart and down would otherwise report success against a project with nothing in it.
+test_env_require_containers() {
+  local containers
+  containers="$(test_env_compose ps -a -q)"
+  if [[ -z "$containers" ]]; then
+    echo "error: Compose project '$TEST_ENV_PROJECT' has no containers." >&2
+    echo "       'docker compose ls -a' lists the projects that exist; a stack this checkout" >&2
+    echo "       started under another name needs CHT_TEST_ENV_PROJECT=<name>." >&2
+    exit 1
+  fi
+  test_env_require_owner
+  return 0
+}

scripts/test-env-down.sh

@@ -0,0 +1,17 @@
+#!/usr/bin/env bash
+#
+# Human-run teardown for the Test Environment Layer.
+# The cht-agent NEVER runs this or any Docker command itself.
+# Usage: scripts/test-env-down.sh [<cht-core-path>]   (overrides: scripts/lib/test-env.sh)
+set -euo pipefail
+source "$(dirname "$0")/lib/test-env.sh"
+
+# Same resolution as test-env-up.sh, minus the clone: the stack must already exist.
+TARGET="$(test_env_default_target "${1:-}")"
+test_env_require_build "$TARGET"
+test_env_select "$TARGET"
+test_env_require_containers
+test_env_compose down -v
+
+echo "CHT environment '$TEST_ENV_PROJECT' torn down. -v removed its named volumes; CouchDB data in the" \
+  "${COUCHDB_DATA:-$TEST_ENV_TARGET/local-build/srv-$TEST_ENV_PROJECT} bind mount stays."

scripts/test-env-restart.sh

@@ -0,0 +1,16 @@
+#!/usr/bin/env bash
+#
+# Human-run restart for the Test Environment Layer (reset tier: restart).
+# The cht-agent NEVER runs this or any Docker command itself.
+# Usage: scripts/test-env-restart.sh [<cht-core-path>]   (overrides: scripts/lib/test-env.sh)
+set -euo pipefail
+source "$(dirname "$0")/lib/test-env.sh"
+
+# Same resolution as test-env-up.sh, minus the clone: the stack must already exist.
+TARGET="$(test_env_default_target "${1:-}")"
+test_env_require_build "$TARGET"
+test_env_select "$TARGET"
+test_env_require_containers
+test_env_compose restart
+
+echo "CHT services in '$TEST_ENV_PROJECT' restarted. The agent should re-confirm health (provision/waitForReady)."

scripts/test-env-up.sh

@@ -0,0 +1,110 @@
+#!/usr/bin/env bash
+#
+# Human-run bring-up for the Test Environment Layer (Model A — rebuild on change).
+# The cht-agent NEVER runs this or any Docker command itself; the agent's
+# provision() only polls /api/v2/monitoring and waits for this to finish. The
+# clone/install below is this HUMAN script's convenience — the agent still never
+# clones, installs, or runs Docker.
+#
+# Usage: scripts/test-env-up.sh [<cht-core-path>]
+#
+# With no argument the stack is built from $CHT_CORE_PATH, or from a managed
+# checkout at $CHT_CORE_CLONE_DIR (default <repo>/.cht-core), cloned from master
+# on first use. A path you pass is used as-is and is never cloned into.
+#
+# Project, credential, port and cert overrides: see scripts/lib/test-env.sh.
+#
+# TLS: the stack serves a SAN-less self-signed cert whose CN is COMMON_NAME (nginx).
+# The agent's cht-conf child accepts it via --accept-self-signed-certs; the agent's
+# own fetch (readiness/discovery/reset) needs it trusted via NODE_EXTRA_CA_CERTS
+# (copy command printed at the end). NODE_TLS_REJECT_UNAUTHORIZED=0 disables
+# verification for ALL of the agent's traffic (LLM/MCP included) and is acceptable
+# only inside a disposable runner container.
+set -euo pipefail
+source "$(dirname "$0")/lib/test-env.sh"
+
+CLONE_DIR="${CHT_CORE_CLONE_DIR:-$TEST_ENV_REPO_ROOT/.cht-core}"
+CHT_CORE_UPSTREAM="${CHT_CORE_UPSTREAM:-https://github.com/medic/cht-core.git}"
+CHT_CORE_BRANCH="${CHT_CORE_BRANCH:-master}"
+
+# cht-core's own engines require Node >= 22.15; npm ci and the image build both
+# fail in confusing ways on older runtimes.
+NODE_MAJOR="$(node -v 2>/dev/null | sed 's/^v\([0-9]*\).*/\1/')"
+if [[ -z "$NODE_MAJOR" ]] || [[ "$NODE_MAJOR" -lt 22 ]]; then
+  echo "error: cht-core needs Node >= 22.15 (found: $(node -v 2>/dev/null || echo 'no node')). Try 'nvm use 22'." >&2
+  exit 1
+fi
+
+TARGET="${1:-${CHT_CORE_PATH:-}}"
+if [[ -z "$TARGET" ]]; then
+  TARGET="$CLONE_DIR"
+  if [[ ! -d "$TARGET/.git" ]]; then
+    echo "No cht-core path given — cloning $CHT_CORE_BRANCH into $TARGET (shallow)."
+    echo "Pass a path, or set CHT_CORE_PATH, to build from a working copy instead."
+    # Shallow is safe: cht-core derives its image version from the branch name
+    # (cht-core's scripts/build/versions.js), not from git tags.
+    git clone --depth 1 --branch "$CHT_CORE_BRANCH" "$CHT_CORE_UPSTREAM" "$TARGET"
+  fi
+fi
+
+if [[ ! -d "$TARGET" ]]; then
+  echo "error: cht-core path not found: $TARGET" >&2
+  exit 1
+fi
+test_env_select "$TARGET"
+test_env_require_owner
+
+# 1. Shared network the cht-agent and CHT both join.
+docker network inspect "$TEST_ENV_NETWORK" >/dev/null 2>&1 || docker network create "$TEST_ENV_NETWORK" >/dev/null
+
+# The agent reaches CHT as https://nginx on that network; a second stack's nginx
+# there would split the name between two instances. -a: a stopped nginx comes back
+# when the Docker daemon restarts (restart: always).
+nginx_projects="$(docker ps -a --filter "network=$TEST_ENV_NETWORK" --filter label=com.docker.compose.service=nginx \
+  --format '{{.Label "com.docker.compose.project"}}')"
+while IFS= read -r project; do
+  if [[ -n "$project" && "$project" != "$TEST_ENV_PROJECT" ]]; then
+    echo "error: Compose project '$project' already has an nginx on $TEST_ENV_NETWORK." >&2
+    echo "       Tear it down first: CHT_TEST_ENV_PROJECT='$project' scripts/test-env-down.sh <its cht-core path>" >&2
+    echo "       If that checkout is gone: docker rm -f \$(docker ps -aq --filter label=com.docker.compose.project=$project)" >&2
+    exit 1
+  fi
+done <<< "$nginx_projects"
+
+# `npm run local-images` builds from node_modules (bowser, uglifyjs, cleancss);
+# without them it dies on an opaque `cp: cannot stat` deep inside the build.
+if [[ ! -d "$TARGET/node_modules" ]]; then
+  echo "Installing cht-core dependencies in $TARGET (npm ci — several minutes, ~1.2GB)."
+  # Lifecycle scripts must run: cht-core's postinstall is patch-package, and skipping
+  # it leaves the patches unapplied and the build broken. This is why --ignore-scripts
+  # is not used here. The code being installed is the same tree we are about to build
+  # images from and run, so npm scripts add no privilege beyond what follows.
+  ( cd "$TARGET" && npm ci )
+elif [[ ! -e "$TARGET/node_modules/bowser/bundled.js" ]] || [[ ! -x "$TARGET/node_modules/.bin/uglifyjs" ]]; then
+  echo "error: cht-core dependencies in $TARGET look incomplete (the image build needs" >&2
+  echo "       bowser + uglifyjs). Run 'npm ci' there yourself — this script will not" >&2
+  echo "       replace an existing node_modules." >&2
+  exit 1
+fi
+
+# 2. Build the app. `npm run local-images` only PACKAGES an already-built tree:
+# build-service-images.sh copies into api/build/static/, which is created by
+# build-prepare.sh (ddocs, enketo css, admin app) and filled by build-webapp-dev.
+# npm ci alone does not produce it — build-dev also runs the per-module installs.
+if [[ ! -d "$TARGET/api/build/static" ]] || [[ "${CHT_CORE_REBUILD:-}" = "1" ]]; then
+  echo "Building cht-core in $TARGET (npm run build-dev — several minutes)."
+  ( cd "$TARGET" && npm run build-dev )
+else
+  echo "Reusing the existing cht-core build in $TARGET."
+  echo "Set CHT_CORE_REBUILD=1 to rebuild after changing cht-core source (Model A is rebuild-on-change)."
+fi
+
+# 3. Package those build outputs into local Docker images.
+( cd "$TARGET" && npm run local-images )
+
+# 4. Start the stack, joined to the shared network via the override.
+test_env_compose up -d
+
+echo "CHT starting as Compose project '$TEST_ENV_PROJECT' on '$TEST_ENV_NETWORK'. The agent will poll /api/v2/monitoring until healthy."
+echo "To trust its cert: docker cp $TEST_ENV_PROJECT-nginx-1:/etc/nginx/private/cert.pem <file>, then NODE_EXTRA_CA_CERTS=<file> for the agent."
+echo "Node reads that file once per process: start the agent after this cert exists, and restart it after test-env-down.sh (the next up makes a new cert)."