Skip to content

feat(infra): add a capped staging stack on the production box - #841

Open
anneoneone wants to merge 1 commit into
dev/flowfrom
feat/840-staging-on-prod-box
Open

anneoneone wants to merge 1 commit into
dev/flowfrom
feat/840-staging-on-prod-box

Conversation

@anneoneone

Copy link
Copy Markdown
Collaborator

Adds a staging stack for dev/flow images at https://staging.live.moafunk.de. For now it runs on the production box as a separate compose project with hard CPU and memory caps. It moves to its own server later. Nothing changes for production until someone runs the setup on the box.

Issue: #840

Validation:

  • bash backend/staging/staging.test.sh (macOS bash 3.2): 19 of 19 pass. Every unsafe setting is refused, the safe one passes, and the untouched example is refused. The new staging-tests.yml runs it on Ubuntu.
  • docker compose config (with STAGING_TAG=sha-16c560e): valid. Caps resolve to API 0.75 CPU / 640 MB, Liquidsoap 0.5 / 320 MB, Icecast 0.25 / 96 MB, all restart: "no", API only on 127.0.0.1:8001. Without a sha-* tag, compose refuses to start.
  • nginx -t on nginx-staging.conf (nginx 1.27): ok.
  • icecast.staging.xml: well-formed XML.
  • Not run: setup, up, seed-db against the real box. My SSH to the production box is blocked, so Anton runs them.
Implementation and review

Not architecture-epic work, so the epic fields do not apply.

Epic: n/a
Executor: n/a
Lane: n/a
Reviewer: n/a
Leaf IDs: n/a

What staging.sh checks before it starts anything

  • .env: APP_ENV=staging, ADMIN_BASE_URL = the staging host, SOUNDCLOUD_AUTO_UPLOAD=false (the default is on), TELEGRAM_ENABLED=false, both bucket names are staging buckets (unset names fall back to moafunk-prod), producer and status URLs point at the staging harbor 8015 and Icecast 8020. No GITHUB_DISPATCH_TOKEN (it would start the production backup workflow), SoundCloud, Telegram, Instagram, Facebook, OpenAI or RECOVERY_* keys.
  • dotenvy traps: duplicate keys (dotenvy keeps the first), unquoted values with a space (dotenvy skips every later line), leftover <placeholders>.
  • stream.env: ports 8015/8020, all passwords set, none equal to a production password from /etc/moafunk/stream.env (values are never printed).
  • Production idle: /api/stream/status says not active, and no show is live or starts within 2 hours (read-only query of the production DB).
  • Resources: at least 10 GB free disk and 1.5 GB available memory; ports 8001/8015/8020 free unless staging already runs.

Isolation

  • Own SQLite DB under /opt/unheard-staging/data. seed-db copies production with the SQLite backup API, then deletes sessions, deletes app_settings rows with token or secret keys (the SoundCloud token lives there) and makes every password unusable.
  • Own Liquidsoap and Icecast containers. They reuse production's moafunk.liq unchanged (it reads its ports from the environment) and the existing moafunk/icecast-kh:kh22 image (pull_policy: never). Icecast's health check is overridden to port 8020.
  • The nginx vhost is installed only after nginx -t passes, is never overwritten on re-run (certbot edits it), and gets TLS only when DNS already points at the box.

Box steps (Anton or phaabe): DNS A record, R2 buckets plus a token scoped to them, Bitwarden STAGING_* values, copy the files, staging.sh setup, optionally seed-db, staging test accounts. All in backend/staging/README.md.

Goal: #829.

Staging runs dev/flow images at staging.live.moafunk.de before a release
reaches production. For now it shares the production box as a separate
compose project with hard CPU and memory caps; it moves to its own
server later.

staging.sh refuses to start when a setting could reach production or
real accounts (buckets, harbor, SoundCloud, GitHub dispatch, Telegram,
Instagram, OpenAI, recovery mail), when production is on air or a show
starts within 2 hours, or when disk or memory is low. seed-db makes a
copy of the production DB without sessions, stored tokens or usable
passwords. staging.test.sh covers every refusal and runs in CI.

Refs: #840
@vercel

vercel Bot commented Nov 3, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
live-moafunk-de Ready Ready Preview Nov 3, 2026 10:03am UTC

This branch was successfully deployed

1 active deployment
Preview — b516dbbe Deployed Nov 3, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant