Linux workers: automated vault.yaml fetch via mTLS

[rcurranmoz]

Replaces the operator's manual vault.yaml SCP step with a step-ca-issued client cert + mTLS fetch from a brokered Secret Manager. Companion to mozilla-platform-ops/relops-bootstrap.

Same mechanism as the macOS Path C work that landed on m4-81 last week — adapted for Linux (no MDM equivalent, so cert minting moves to the operator's laptop instead of via SimpleMDM SCEP profile).

What changes

provisioners/linux/bootstrap_linux.sh (worker-side)

Adds a fetch_vault_yaml_via_mtls function that:

• Looks for a client cert at /etc/relops-bootstrap/{cert,key}.pem
• If present, curl --cert ... --key ... to https://forge.relops.mozilla.com/secret/<role>
• Atomic-writes the response to /root/vault.yaml
• Hands off to the existing puppet flow unchanged

If the cert isn't staged, the function is a no-op and falls through to the existing "vault.yaml must be hand-dropped" check. Existing workers being provisioned today via the legacy flow continue to work unchanged.

provisioners/linux/deliver_linux.sh (operator-side)

Adds opt-in mTLS path gated on RELOPS_BOOTSTRAP_REPO env var:

• When set, runs $RELOPS_BOOTSTRAP_REPO/scripts/mint-scep-cert.sh to mint a fresh cert+key for the target host+role via SCEP against step-ca (using the operator's gcloud auth as the gate on the SCEP challenge)
• SCPs cert+key to /etc/relops-bootstrap/ on the worker (root:root 0600)
• Operator no longer needs a local vault.yaml file

When RELOPS_BOOTSTRAP_REPO is unset, the original flow is preserved.

Architecture

Full design doc in mozilla-platform-ops/relops-bootstrap. Quick version:

• step-ca (GCE VM) has per-role SCEP provisioners (scep-gecko-t-linux-2404-talos, etc.). Each provisioner's x509 template hardcodes the puppet role into the cert's SPIFFE URI SAN (spiffe://relops.mozilla/host/<CN>/role/<role>).
• GCP HTTPS LB at forge.relops.mozilla.com does mTLS termination against a Trust Config containing the step-ca root + intermediate, forwards X-Client-Cert-Leaf (+ presence/chain-verified flags) to the broker.
• Broker (Cloud Run, FastAPI) parses the leaf cert, URL-decodes the SPIFFE URI from the SAN, role-binds against the URL path, returns vault-<role> from Secret Manager. Per-cert-serial rate limit + Cloud Audit logging.

Testing — @aerickson would you mind taking this for a spin?

Validated end-to-end on macOS hardware already (m4-81 post-EACS). The Linux side hasn't been tested against a real worker yet. Want a second pair of eyes + a real-host run before we merge.

Steps:

# 1. Install prerequisites on your laptop
brew install sscep                # or apt install sscep on Linux
gcloud auth login                  # for Secret Manager access
gcloud config set project relops-bootstrap

# 2. Clone relops-bootstrap (the operator-side helpers live there)
git clone git@github.com:mozilla-platform-ops/relops-bootstrap.git ~/git/relops-bootstrap

# 3. Populate the vault secret for the role you want to test (one-time, from 1P)
op read "op://RelOps Vault/vault-gecko_t_linux_2404_talos/notesPlain" \
  | gcloud secrets versions add vault-gecko_t_linux_2404_talos --data-file=- \
    --project=relops-bootstrap

# 4. Check out this PR branch
cd ~/git/ronin_puppet
git fetch && git checkout rcurran/relops-bootstrap-mtls-vault-fetch

# 5. Run deliver_linux with the new flow against a TEST host
export RELOPS_BOOTSTRAP_REPO=~/git/relops-bootstrap
cd provisioners/linux
./deliver_linux.sh <test-linux-host> gecko_t_linux_2404_talos

# 6. SSH to the worker and kick the bootstrap (with -l to capture log to a file)
ssh root@<test-linux-host> "bash /tmp/bootstrap.sh -l /var/log/bootstrap.log"

What you should see:

• deliver_linux.sh mints a cert via sscep and SCPs it to /etc/relops-bootstrap/{cert,key}.pem on the worker
• bootstrap_linux.sh runs fetch_vault_yaml_via_mtls first, finds the cert, curls to https://forge.relops.mozilla.com/secret/gecko_t_linux_2404_talos, gets HTTP 200 with the vault.yaml content
• File lands at /root/vault.yaml and the rest of the bootstrap proceeds normally

Available Linux roles (per-role step-ca SCEP provisioners are already set up):

• gecko_t_linux_talos
• gecko_t_linux_2204_talos
• gecko_t_linux_2404_talos
• gecko_t_linux_2404_talos_wayland
• gecko_t_linux_netperf
• gecko_t_linux_2404_netperf

(Each role's vault-<role> secret in Secret Manager needs to be populated once before the first worker of that role can fetch — step 3 above. Currently all 6 secret containers exist but have no version data yet.)

If anything goes wrong, the diagnostic trail is:

• bootstrap_linux.sh output: wherever you pointed -l (or stdout if interactive)
• mint-scep-cert.sh errors print to stderr on your laptop
• Broker logs: gcloud logging read 'resource.type=cloud_run_revision AND resource.labels.service_name=vault-broker' --project relops-bootstrap --freshness 5m

[rcurranmoz]

@aerickson would you be up for testing the new Linux Path C flow on a spare Linux test host? Test steps are in the updated PR description above — TL;DR:

1. brew install sscep + gcloud auth login --project relops-bootstrap
2. Clone https://github.com/mozilla-platform-ops/relops-bootstrap alongside this checkout
3. Populate one Linux role's vault secret from 1Password into Secret Manager (vault-gecko_t_linux_2404_talos or whichever role your test host uses)
4. From this branch: export RELOPS_BOOTSTRAP_REPO=~/git/relops-bootstrap and run ./deliver_linux.sh <test-host> <role> — the script now mints + SCPs a step-ca cert instead of requiring a local vault.yaml
5. SSH to the worker and run the bootstrap as usual

The macOS side of this (forge.relops.mozilla.com + broker + step-ca) is already running and proven on m4-81. The Linux client path itself just hasn't been validated against a real Linux worker yet — that's what this test confirms.

Backwards-compat is preserved: don't set RELOPS_BOOTSTRAP_REPO and deliver_linux.sh behaves identically to today.

[aerickson]

The diff is cleanly backward-compatible, but the new Linux path introduces a custom cert/broker dependency for what used to be a direct vault.yaml copy. This seems like a clear win for the macOS infrastructure / DEP flow. For Linux specifically, the operator still needs privileged local authority to mint the bootstrap cert, so the cert-minting step itself is not a security improvement. For Linux, I think the main security upside would be narrower role-scoped secret delivery, but that part is somewhat separate from the cert/broker mechanism itself (could be done in 1p also).

• The staged cert/key are left on the worker in /etc/relops-bootstrap. If they are only needed once to fetch /root/vault.yaml, should they be deleted after a successful fetch?
• /root/vault.yaml still persists as the final artifact, so this changes the delivery path but not the on-host secret-at-rest model.
• bootstrap_linux.sh does not validate that the broker response is non-empty or valid YAML before moving it into place as /root/vault.yaml.

I’ll test it out.