feat(infra): add a capped staging stack on the production box

.github/workflows/staging-tests.yml

@@ -0,0 +1,30 @@
+name: Staging safety checks
+
+# staging.sh must refuse every staging setting that could reach production or
+# real accounts. No secrets, no box access: the test uses throwaway env files.
+on:
+  pull_request:
+    branches: [main, dev/flow]
+    paths:
+      - backend/staging/**
+      - .github/workflows/staging-tests.yml
+  push:
+    branches: [main, dev/flow]
+    paths:
+      - backend/staging/**
+      - .github/workflows/staging-tests.yml
+
+permissions:
+  contents: read
+
+jobs:
+  staging-tests:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          persist-credentials: false
+
+      - name: Check staging.sh refuses unsafe settings
+        run: bash backend/staging/staging.test.sh

backend/staging/README.md

@@ -0,0 +1,54 @@
+# Staging
+
+Staging runs `dev/flow` images at https://staging.live.moafunk.de before a release reaches `main` and production.
+
+**Stopgap:** staging runs on the production box, as a separate and capped compose project. It moves to its own small server later (a `cx23` costs about €5.49 per month). Until then, follow the rules below exactly.
+
+## Rules
+
+- Staging never reaches production data or real accounts. `staging.sh check-env` enforces this:
+  - own SQLite DB (a scrubbed copy of production, see `seed-db`)
+  - own R2 buckets (`unheard-artists-staging`, `moafunk-staging`) with a token that can only reach them
+  - own Liquidsoap and Icecast (harbor 8015, Icecast 8020) with own passwords
+  - `SOUNDCLOUD_AUTO_UPLOAD=false`, `TELEGRAM_ENABLED=false`
+  - no `GITHUB_DISPATCH_TOKEN` (it would start the production backup workflow), no SoundCloud, Telegram, Instagram, Facebook, OpenAI or `RECOVERY_*` settings
+- Staging runs only while someone tests, and never during a show. `staging.sh up` refuses when production is on air, when a show starts within 2 hours, or when the box has less than 10 GB disk or 1.5 GB memory free.
+- Hard caps: API 0.75 CPU / 640 MB, Liquidsoap 0.5 CPU / 320 MB, Icecast 0.25 CPU / 96 MB. Nothing restarts on its own, not even after a reboot.
+- Never start or stop staging containers by hand; always use `staging.sh`.
+
+## One-time setup
+
+Done by someone with access to the box, DNS, Cloudflare and Bitwarden.
+
+1. DNS: an A record `staging.live.moafunk.de` → `178.104.160.103` (Hetzner DNS zone `moafunk.de`).
+2. Cloudflare R2: create the buckets `unheard-artists-staging` and `moafunk-staging`, and an API token that can only read and write those two buckets.
+3. Bitwarden: store the `STAGING_*` values from `staging.env.example` and `stream.env.example`. Generate new passwords; never reuse production ones.
+4. Copy the files to the box:
+
+   ```bash
+   ssh root@178.104.160.103 'mkdir -p /opt/unheard-staging'
+   scp backend/staging/{docker-compose.staging.yml,icecast.staging.xml,nginx-staging.conf,staging.sh} docs/stream-rework/prod/moafunk.liq root@178.104.160.103:/opt/unheard-staging/
+   ```
+
+5. On the box, create `/opt/unheard-staging/.env` and `/opt/unheard-staging/stream.env` from the two examples, with the Bitwarden values.
+6. On the box: `CERTBOT_EMAIL=<email> /opt/unheard-staging/staging.sh setup`. It checks both env files, writes `icecast.xml`, installs the nginx vhost (after `nginx -t`) and gets the TLS certificate once DNS points at the box.
+7. Optional: `/opt/unheard-staging/staging.sh seed-db` for a copy of production data. The copy has no sessions, no stored tokens and no usable passwords. Shows from production can show up as missed in staging logs; that is expected.
+8. Log in as the staging superadmin and create staging test accounts (a host and an admin).
+
+## Daily use
+
+Pushes to `dev/flow` build an image `sha-<7 chars>` (`backend.yml`). Then, on the box:
+
+```bash
+/opt/unheard-staging/staging.sh up sha-1234567   # checks, pulls, starts, waits for healthy
+/opt/unheard-staging/staging.sh status
+/opt/unheard-staging/staging.sh down             # always stop when done
+```
+
+## Tests
+
+`bash backend/staging/staging.test.sh` checks that every unsafe setting stops staging. It needs no Docker and no box access.
+
+## Removing staging
+
+`staging.sh down`, then delete `/opt/unheard-staging`, the nginx vhost `staging.live.moafunk.de` (and its certificate), the two R2 buckets and the DNS record.

backend/staging/docker-compose.staging.yml

@@ -0,0 +1,68 @@
+# Staging stack on the production box (a stopgap until staging gets its own
+# server). Separate project, ports and data; hard CPU/memory caps so staging
+# can never starve production. Nothing restarts on its own: staging runs only
+# while someone tests. Start and stop it with staging.sh, never by hand.
+name: unheard-staging
+
+services:
+  api:
+    image: "ghcr.io/phaabe/live.moafunk.de-backend:${STAGING_TAG:?set STAGING_TAG to a sha-* tag}"
+    container_name: unheard-api-staging
+    restart: "no"
+    ports:
+      - "127.0.0.1:8001:8000"
+    # Same as production: the API is bridge-networked, the stream chain is
+    # host-networked, so the API reaches the staging harbor via the host gateway.
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - ./data:/app/data
+      - ./.env:/app/.env:ro
+    environment:
+      - TZ=Europe/Berlin
+    cpus: "0.75"
+    mem_limit: 640m
+    memswap_limit: 640m
+    pids_limit: 256
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 10s
+
+  # Production's moafunk.liq, unchanged: it reads its ports from stream.env
+  # (harbor 8015, Icecast 8020 here; production uses 8005 and 8010).
+  liquidsoap:
+    image: savonet/liquidsoap:v2.4.4
+    pull_policy: missing
+    container_name: moafunk-liquidsoap-staging
+    restart: "no"
+    network_mode: host
+    env_file: ./stream.env
+    volumes:
+      - ./moafunk.liq:/moafunk.liq:ro
+    command: ["liquidsoap", "/moafunk.liq"]
+    cpus: "0.5"
+    mem_limit: 320m
+    memswap_limit: 320m
+
+  # The production Icecast-KH image (built on the box, not in a registry).
+  icecast:
+    image: moafunk/icecast-kh:kh22
+    pull_policy: never
+    container_name: moafunk-icecast-staging
+    restart: "no"
+    network_mode: host
+    volumes:
+      - ./icecast.xml:/etc/moafunk/icecast.xml:ro
+    cpus: "0.25"
+    mem_limit: 96m
+    memswap_limit: 96m
+    # The image's own health check probes production's port 8010.
+    healthcheck:
+      test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8020/status-json.xsl"]
+      interval: 30s
+      timeout: 5s
+      retries: 3
+      start_period: 10s

backend/staging/icecast.staging.xml

@@ -0,0 +1,55 @@
+<!-- Staging Icecast-KH config. Same shape as docs/stream-rework/prod/icecast.xml,
+     but port 8020, small limits and nothing listed publicly. staging.sh setup
+     fills the CHANGE_ME passwords from stream.env into
+     /opt/unheard-staging/icecast.xml (mode 600). Port 8020 is reached only
+     through nginx and localhost; the cloud firewall allows 22/80/443 only. -->
+<icecast>
+  <location>Berlin</location>
+  <admin>admin@moafunk.de</admin>
+
+  <limits>
+    <clients>20</clients>
+    <sources>4</sources>
+    <queue-size>524288</queue-size>
+    <burst-size>65536</burst-size>
+  </limits>
+
+  <authentication>
+    <source-password>CHANGE_ME_SOURCE</source-password>
+    <relay-password>CHANGE_ME_RELAY</relay-password>
+    <admin-user>admin</admin-user>
+    <admin-password>CHANGE_ME_ADMIN</admin-password>
+  </authentication>
+
+  <listen-socket>
+    <port>8020</port>
+  </listen-socket>
+
+  <mount type="normal">
+    <mount-name>/live.mp3</mount-name>
+    <max-listeners>10</max-listeners>
+    <public>0</public>
+  </mount>
+
+  <mount type="normal">
+    <mount-name>/test.mp3</mount-name>
+    <max-listeners>10</max-listeners>
+    <public>0</public>
+  </mount>
+
+  <fileserve>1</fileserve>
+
+  <paths>
+    <logdir>/var/log/icecast</logdir>
+    <webroot>/usr/local/share/icecast/web</webroot>
+    <adminroot>/usr/local/share/icecast/admin</adminroot>
+  </paths>
+
+  <logging>
+    <loglevel>2</loglevel>
+  </logging>
+
+  <security>
+    <chroot>0</chroot>
+  </security>
+</icecast>

backend/staging/nginx-staging.conf

@@ -0,0 +1,54 @@
+# nginx vhost for staging. staging.sh setup installs it as
+# /etc/nginx/sites-available/staging.live.moafunk.de; certbot then adds TLS.
+# Mirrors the production vhost in backend/scripts/deploy_hetzner.sh, but points
+# at the staging API (127.0.0.1:8001) and the staging Icecast (127.0.0.1:8020).
+server {
+  listen 80;
+  server_name staging.live.moafunk.de;
+  client_max_body_size 260m;
+
+  location = /metrics {
+    return 404;
+  }
+
+  location = /live.mp3 {
+    proxy_pass http://127.0.0.1:8020/live.mp3;
+    proxy_http_version 1.1;
+    proxy_buffering off;
+    proxy_set_header Host $host;
+    proxy_read_timeout 3600s;
+    proxy_hide_header Access-Control-Allow-Origin;
+    add_header Access-Control-Allow-Origin "*" always;
+  }
+
+  location = /test.mp3 {
+    proxy_pass http://127.0.0.1:8020/test.mp3;
+    proxy_http_version 1.1;
+    proxy_buffering off;
+    proxy_set_header Host $host;
+    proxy_read_timeout 3600s;
+    proxy_hide_header Access-Control-Allow-Origin;
+    add_header Access-Control-Allow-Origin "*" always;
+  }
+
+  location = /status-json.xsl {
+    proxy_pass http://127.0.0.1:8020/status-json.xsl;
+    proxy_http_version 1.1;
+    proxy_set_header Host $host;
+    add_header Access-Control-Allow-Origin "*" always;
+  }
+
+  location / {
+    proxy_pass http://127.0.0.1:8001;
+    proxy_http_version 1.1;
+    proxy_set_header Upgrade $http_upgrade;
+    proxy_set_header Connection 'upgrade';
+    proxy_set_header Host $host;
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+    proxy_set_header X-Forwarded-Proto $scheme;
+    proxy_connect_timeout 300;
+    proxy_send_timeout 300;
+    proxy_read_timeout 300;
+  }
+}

backend/staging/staging.env.example

@@ -0,0 +1,38 @@
+# Staging API settings. Copy to /opt/unheard-staging/.env and fill the
+# <...> values from Bitwarden (STAGING_* items). Never copy production values.
+# staging.sh refuses to start when a value below is unsafe.
+#
+# dotenvy stops reading at the first unquoted value with a space and skips
+# every later line, so quote values like "Moafunk Staging".
+
+APP_ENV=staging
+ADMIN_BASE_URL=https://staging.live.moafunk.de
+
+SECRET_KEY=<STAGING_SECRET_KEY>
+SUPERADMIN_USERNAME=<STAGING_SUPERADMIN_USERNAME>
+# Argon2 hash from `cargo run --bin hash_password -- '<password>'`. Single
+# quotes: dotenvy expands `$` inside double quotes.
+SUPERADMIN_PASSWORD_HASH='<STAGING_SUPERADMIN_PASSWORD_HASH>'
+
+# Own buckets, and a token that can only reach them. Unset bucket names fall
+# back to production buckets, so both lines are required.
+R2_ACCOUNT_ID=<R2_ACCOUNT_ID>
+R2_ACCESS_KEY_ID=<STAGING_R2_ACCESS_KEY_ID>
+R2_SECRET_ACCESS_KEY=<STAGING_R2_SECRET_ACCESS_KEY>
+R2_BUCKET_NAME=unheard-artists-staging
+R2_SHOWS_BUCKET_NAME=moafunk-staging
+
+# Own stream chain: staging harbor 8015, staging Icecast 8020.
+STREAM_OUTPUT=icecast
+ICECAST_URL=icecast://source:<STAGING_HARBOR_LIVE_PASSWORD>@host.docker.internal:8015/live
+ICECAST_TEST_URL=icecast://source:<STAGING_HARBOR_TEST_PASSWORD>@host.docker.internal:8015/test
+ICECAST_STATUS_URL=http://host.docker.internal:8020/status-json.xsl
+
+# Never reach real accounts from staging.
+SOUNDCLOUD_AUTO_UPLOAD=false
+TELEGRAM_ENABLED=false
+
+# Must stay absent in staging (staging.sh checks):
+# GITHUB_DISPATCH_TOKEN (would start the production backup workflow),
+# SOUNDCLOUD_*, TELEGRAM_BOT_TOKEN, TELEGRAM_*_CHAT_ID, INSTAGRAM_*,
+# FACEBOOK_*, OPENAI_API_KEY, RECOVERY_*.