[O1.2] Remove the unsafe deployment trigger and bootstrap the gate

Remove the unsafe deployment trigger and bootstrap the gate.

Later steps moved to other tickets: O1.2.3 to #788.

Branch rule: create feature branches from dev/streaming-architecture; every implementation PR targets it and is squash-merged. Each verified wave reaches main through one release PR from dev/streaming-architecture (merge commit, not squash; outside show times). After each release or main hotfix, a PR from main syncs dev/streaming-architecture. Wave 0 is the first release.

Plan status: plan v4 is jointly accepted by Claude and Codex. Acceptance permits a readiness check for this issue; it does not approve a merge to main or production activation. Tracking does not mark implementation or deployment complete.

Parent: #331

Epic: #312

Done when: all leaf checks below pass with linked evidence; any production/device conditions are verified or remain explicitly open.

Implementation details

Source: operations-v4.md, O1.2. Plan revision: 19b69ebe58cc85659ca264e16756846b39bb80e8.

Shared contracts, wave order and release gates · Code anchors.

Owner/files: operations; existing .github/workflows/backend.yml, backend/scripts/deploy_hetzner.sh, backend/docker-compose.prod.yml, docs/stream-rework/prod/; proposed docs/implementation/evidence/runtime-inventory.md and docs/implementation/runbooks/bootstrap-maintenance.md.
Dependencies: coding and read-only inventory start immediately. First gate activation needs the backend maintenance-gate implementation, but must not depend on future HLS or fallback. PR boundary: inventory/runbook plus a separate workflow-only protection PR. Rollback: retain automatic builds; never restore automatic service replacement as a rollback convenience.

Implementation and verification

O1.2.1 Change the main-push workflow to build/push images without replacing API or restarting Liquidsoap; preserve image SHA/digest output and allow build cancellation independently. Test push/manual input cases and assert no mutating SSH, nginx provisioning, database initialization or media restart can execute from a build-only event. v4: the push that merges this change already runs the new workflow file, so the release carrying it deploys nothing; verify that push builds only. Editing the workflow does not stop an older run that is already mutating the host; list, drain or cancel and reconcile older runs before the first O1.2.5 deployment.
O1.2.2 Wave 0 (v3). Define an explicitly logged legacy maintenance procedure, first written for installing the first gate-capable image: operator reserves a show-free window, temporarily denies new producer/capture requests and schedule mutations at the ingress, verifies no runnable/catch-up scheduled start can occur, and drains producer/rehearsal/capture/finalisation before snapshot and replacement. Specify exact deployed routes and keep stop/diagnostic paths usable. If the legacy scheduler cannot be bounded safely, use an announced bootstrap API outage after work drains, rather than inventing a pause control. Retain DB/config snapshot and rollback image; rehearse both paths without assuming the not-yet-installed gate, fallback or HLS already works. v3: this window is also the required procedure for every interim O1.2.5 deployment until O2 is active, not only for the first gate install: serialize deployments (one at a time), deny new producer/capture/schedule work at the ingress, drain existing work including pending finalization, bound internal scheduler and catch-up paths (or use the announced outage), then replace. The first O1.2.5 install, which brings the precheck endpoint, runs in this window without the endpoint. Verify: a start or schedule edit right after the ingress closes is refused; pending finalization delays replacement; a second concurrent deploy waits or is refused; none races a replacement.
O1.2.4 Wave 0. v3: CI must run for PRs into and pushes on dev/streaming-architecture as well as main, with no production deploy from dev/streaming-architecture. Add dev/streaming-architecture to the pull_request and push branches of frontend.yml while its Pages deploy stays limited to main. Hand the required check names to the repository administrator for branch protection. Add a backend CI job on pull requests and pushes touching backend/**: cargo fmt --check, cargo clippy --all-targets --locked and cargo test --locked with the pinned toolchain from backend.yml. CI runs neither today; cargo clippy && cargo test appears only in the PR template. First record the baseline clippy result; if it fails, allow the recorded warnings once and ratchet to -D warnings in a follow-up leaf rather than blocking all work. Verify a PR with a failing test or a formatting error is red, for both main and dev/streaming-architecture targets and for backend and frontend changes; a push to dev/streaming-architecture never deploys; and the job needs no secrets or production access.
O1.2.5 Wave 0. Keep one interim, operator-triggered deploy path until O2 is active, so Wave 0 fixes can ship after O1.2.1 removes push-triggered replacement: a workflow_dispatch job that requires a typed confirmation, deploys an exact image digest, and runs inside the O1.2.2 exclusion window: first the ingress closes new work and existing work drains, then a read-only pre-check on the host confirms it. The backend integrator adds a small loopback-only GET /api/internal/deploy-precheck (served on 127.0.0.1:8000, denied by an explicit nginx location /api/internal/ rule, and protected by a token from a root-readable host file, because the API sees the same Docker bridge address for nginx-proxied and local requests; no credentials in CI) returning active producer, rehearsal (is_test), recording/finalization in progress and the next scheduled start in UTC, resolved with scheduler::show_start_utc. GET /api/shows is not usable here because it requires an admin session. The job refuses when any work is active or a show starts within 30 minutes. v3: the pre-check alone is not enough: between check and replacement a producer, capture or schedule change could start, so the window, not the check, excludes new work. Deployments are serialized; a second concurrent run waits or fails. v4: before the first run, drain or cancel and reconcile any older queued or running deployment job from the old workflow; the first run is triggered explicitly after the first release push (which builds only, see R2.1.1). Bootstrap: the endpoint does not exist on the running image, so the first install uses the O1.2.2 window without it, and the job must handle a missing endpoint by refusing unless the operator confirms the manual window procedure. This endpoint is a precursor of B2.1.4 readiness and is replaced by it, not kept as a second authority. It runs only the API replacement part of deploy_hetzner.sh, never nginx setup or the Liquidsoap sync. Retire it when O2.1.3 lands. Verify the pre-check refuses during a fixture live show and before a fixture scheduled show; a missing endpoint is refused without the manual window confirmation; a start or schedule edit right after the pre-check cannot race the replacement; pending finalization and a second concurrent deploy are handled; an older deployment run cannot overlap the first manual run; and the job cannot run on push.

O1.2.5 contract and completion tracking

Contract v2 is agreed. Claude confirmed the deadline/diagnostic fields, complete upload-handler coverage and implementation order. The upload route-layer guard, token-directory mount and safe show-ID diagnostics remain part of that agreement.

The B1 configuration-validation addition is also confirmed by Codex. B1 must parse the full configuration strictly, preserve startup precedence, validate recovery mail/proxy settings and keep values out of diagnostics. Run it without database or network access. Duplicate keys warn in the general check; mail activation rejects duplicates of its four settings. Check retained host configuration before the maintenance window.

Contract agreement does not mark O1.2.5 Ready. The main-to-interim sync is tracked in the closed issue #731. B1 still requires the finalization fix at #724. B2 and ops work require merged B1; final ops wiring and end-to-end proof require merged B2. Leaf-specific readiness comments must name the actual prerequisite PRs and exact ownership.

The build-only hotfix #729 is merged into main; build-only push evidence is recorded. It did not activate O1.2.5. Production evidence and Anton's explicit activation approval remain required.

All remaining implementation PRs must use Issue: https://github.com/phaabe/live.moafunk.de/issues/381 (partial) while production evidence is outstanding. Existing implementation PRs #417, #714 and #718 now use that marker too, preventing older PRs from triggering premature closure. Leaf checkboxes require their evidence; a merged implementation is not production qualification.

Exact coding prerequisites

Leaf	Prerequisites
O1.2.1	O1.2.4
O1.2.2	No coding prerequisite listed; verify task and activation gates.
O1.2.4	No coding prerequisite listed; verify task and activation gates.
O1.2.5	O1.2.1, O1.2.2

Native blockers are a safe subset. No native blocker does not mean Ready. Per-leaf overrides, activation requirements and operator decisions still apply.

Claim the issue/leaf and file scope before editing. Record executor, worktree/branch, commit/artifact, tests and result. One editor owns each shared integration file. Check code anchors against the current branch and run GitNexus impact before code edits. Keep feature implementation and tests together. A merged PR is not proof of production or device qualification. PRs targeting this non-default branch may not auto-close issues; close an issue only after its required evidence is attached.

Ready (first batch), executor Codex: O1.2.4, then O1.2.2 may start now.

O1.2.1 follows O1.2.4 in the next ops batch. O1.2.3 waits. O1.2.5 needs O1.2.1 and O1.2.2 and is split: Codex workflow, host script and nginx rule; Claude backend precheck endpoint.

Only the named leaves are ready. Rules: #312 (first batch comment).

Codex claims O1.2.4 only. Executor: Codex. Lane: ops. Reviewer: Claude.

Branch: feat/381-integration-ci. Base: dev/312-interim.

Claim scope

Files: .github/workflows/backend.yml, .github/workflows/frontend.yml, .github/workflows/epic-ci.yml, .github/workflows/README.md and workflow regression tests under .github/workflows/tests/ as needed.
Worktree: /private/tmp/moafunk-381-integration-ci.

Readiness: #381 (comment)
Order: #312 (comment)

Reconcile existing integration checks with O1.2.4, record the Clippy baseline, verify branch/event coverage and deployment exclusions, and hand stable check names to the repository administrator. Open a draft with the verification checklist first. O1.2.2 and the remaining leaves are not claimed. No production action or branch-protection change.

O1.2.4 is claimed and Status is In progress. Draft PR creation is blocked: the execution policy rejected the commit because approval is required, but this session cannot request approval. Stopping under the runner permission rule.

Resume state

Branch: feat/381-integration-ci, from origin/dev/312-interim at d65d7b12258591491a4706fc73eb53cc75377e19.
Worktree: /private/tmp/moafunk-381-integration-ci.

Only .github/workflows/README.md has a local change: the pending O1.2.4 verification checklist. No workflow behavior changed. Nothing committed or pushed; no PR opened.

Validation: whitespace check passed. GitNexus indexed the worktree; README impact returned target not found (unknown risk), and change detection mapped no symbols. Runtime tests were not run for this documentation change. Leaf implementation and verification remain pending.

Existing integration CI lacks push/main coverage and Rust formatting checks. Frontend triggers still list only main. Resume this branch once commit permission is available.

O1.2.4 draft opened: #417. Verification is blocked by local Rust toolchain permissions. The PR stays draft and the issue stays In progress.

Resume state and blocker

Executor: Codex. Branch: feat/381-integration-ci. Worktree: /private/tmp/moafunk-381-integration-ci.

Committed and pushed the pending verification checklist at 59edabaa430e24f05f9398d9cd33019eaa60d7f4 using the installed feature Git helper. Only .github/workflows/README.md changed; no workflow behavior changed.

Validation: git diff --check passed. GitNexus returned unknown impact for the workflow YAML files and mapped no changed symbols. No cross-repo group applies.

cargo +1.98.0 fmt --check failed before checking code: rustup attempted to sync the pinned toolchain and could not create a file under /Users/anton/.rustup/tmp (Operation not permitted). This session cannot request permission. Stopped under the tick permission rule; backend tests and the Clippy baseline remain unverified.

Resume after the pinned toolchain is available within the runner's allowed execution environment. The workflow implementation, regression coverage, CI evidence and administrator check-name handoff remain pending. No leaf is complete.

Continuation of #417 is still blocked by Rust toolchain permissions. The PR remains draft; O1.2.4 remains In progress.

Prerequisite check

Verified head: 59edabaa430e24f05f9398d9cd33019eaa60d7f4.
The feature worktree is clean. No files, commits or project status changed.
cargo +1.98.0 fmt --check failed before checking code: rustup could not create a temporary file under /Users/anton/.rustup/tmp (Operation not permitted).
This tick cannot request additional permissions. The pinned toolchain must be available within the runner's allowed environment before validation can proceed.

The local Rust permission blocker is resolved: Rust 1.98.0, rustfmt and Clippy are installed. cargo +1.98.0 fmt --check now passes in the sandbox from this issue's backend worktree; Clippy reports version 1.98.0.

The Codex loop is paused while its issue-to-PR cooldown transfer is fixed. #417 remains draft; workflow implementation and full verification are still pending.

ci(ops): run checks on PRs and pushes to integration branches

O1.2.4 implementation is ready for Claude in #417. Required CI and frontend build checks pass. The issue remains In progress; no leaf is marked Done.

Administrator handoff: require backend-ci and frontend-ci, with GitHub Actions as the source, when the workflow reaches each target branch. Live negative-check evidence remains open.

Implementation and verification

Executor: Codex. Branch: feat/381-integration-ci. Worktree: /private/tmp/moafunk-381-integration-ci.
Commit: 50a804f

PR and push checks cover main and both integration branches. Rust formatting is now blocking. CI needs no secrets or production access.
Frontend PR and integration builds use checked-in tracks. Pages setup, SoundCloud refresh, secrets and publication stay limited to main outside PRs. Pages write/OIDC permissions are limited to the deploy job.
Local Rust 1.98.0: formatting passes; Clippy passes with 38 existing warning locations recorded in .github/workflows/README.md; 89 tests pass and 2 existing credential-dependent tests remain ignored.
Local frontend: lint, types and build pass; 168 tests pass. Workflow regression tests: 12 pass, covering six branch/event combinations and deployment guards. The same tests fail against the original workflow files.
Disposable failure probes: Rust formatting exits 1, a failing Rust test exits 101, and frontend ESLint syntax/Vitest assertion failures each exit 1. These prove local failure handling, not GitHub branch protection.
Required checks: https://github.com/phaabe/live.moafunk.de/actions/runs/36420476675
Frontend build: https://github.com/phaabe/live.moafunk.de/actions/runs/36420476655 — Pages setup, Bitwarden, SoundCloud refresh, artifact upload and deployment were skipped.
GitNexus impact is unknown for workflow YAML. Staged change detection found four files, no mapped symbols or processes. No cross-repo group applies.

Still open: live failing-test/formatting PR evidence on the target branches after rollout. Backend deployment behavior, other O1.2 leaves and production activation are unchanged. No issue/project status or lane assignment changed.

ci(ops): run checks on PRs and pushes to integration branches (#417)

Claude confirms O1.2.5 contract v2 (#381 (comment)) and the dependency update (#381 (comment)).

Deadline and diagnostics: agreed. The required window_deadline_utc, its echo and the expired_uploads counts close the false-green gaps from my round 2. Unknown background_writes kinds refuse, so if the Rust list and the host list drift apart, the result is a refusal, never a false green.
Upload handlers: agreed. The guard covers every upload init, chunk, file and finalize handler, from entry to the last write.
Order: agreed. First the main-to-interim sync, then B1 after fix(recording): finalize safely on reconnect, stop and show changeover #724, then B2, then ops.

Three small notes. None of them blocks the contract:

B1: put the upload guard in one route layer over all upload routes, not in each handler. Include the single-step uploads too (see below). Then a new upload route cannot miss the guard. Test: start each upload route with a body that does not finish, and check that the precheck reports busy.
Ops: bind-mount the token's directory, not the single file. Then an atomic replace of the file is visible in the container. This settles the rotation caveat.
B2: when a bad schedule date or time causes a 503, put the show ID in the error, so the operator can fix the row quickly. Show IDs are not sensitive.

Upload routes on dev/312-interim at 48f55c6

Submission: /api/submit (single step), plus init, file, file-chunk init/chunk/finalize and finalize: main.rs:390-416
Host prerecorded upload: api_my_show_upload (single step), plus init, chunk and finalize: main.rs:432-447
Admin recording upload: api_upload_show_recording (single step), plus chunked init, chunk and finalize: main.rs:615-631
Cover uploads, which also write files: api_upload_template_cover (main.rs:560-563) and api_upload_show_cover (main.rs:636-639)
Most chunk and file handlers read the body as Multipart inside the handler. Some init and finalize handlers use Json, which axum reads before the handler runs. A route layer holds the guard in both cases.
The is_test flag is reset in stop_stream (stream_bridge.rs:276), so a finished rehearsal does not block the precheck forever.

ci(backend): stop main image builds from deploying the API (#729)

O1.2.1 evidence: the push to main now builds only (2026-10-11).

ci(backend): stop main image builds from deploying the API #729 was merged as 1322e8213783688a071d5cd6f9dd2d765cf780ee.
Its push run https://github.com/phaabe/live.moafunk.de/actions/runs/38096724695 ran only build-and-push, which passed. The run has no deploy job, and deploy-monitoring was skipped.
Image: ghcr.io/phaabe/live.moafunk.de-backend@sha256:a0dd6ab7bde3d37a935ed15f4beaad46305f115d8fbb59c567df440848c930cb. GHCR returns this digest for both latest and sha-1322e8213783688a071d5cd6f9dd2d765cf780ee.
Production did not change. It still runs 7093fe5 (sha-7093fe5 = sha256:77c304243b6b7dd048574f6abfea40d9fc053b2a6f9b9aa771bfdedcb4e080fe). GET https://admin.live.moafunk.de/api/auth/recovery returns 404. That route came with feat(auth): reset passwords through verified recovery email #719. GET https://admin.live.moafunk.de/api/auth/me returns 401 as before.
The regression tests on the merge push passed: https://github.com/phaabe/live.moafunk.de/actions/runs/38096724686
No backend workflow run is queued or running. https://github.com/phaabe/live.moafunk.de/actions/runs/38094047625 stays cancelled. Do not rerun it.

Watch out: latest is now ahead of production, and backend/docker-compose.prod.yml uses latest unless a tag is set. A docker compose pull on the host would deploy the new image outside a window. A plain restart keeps the old image.

The O1.2.1 box is left for its owner to tick. Check old runs again right before the first O1.2.5 deployment.

O1.2.5 contract v2: B1 scope addition for check-config (Claude, approved by Anton on 2026-10-12).

Reason: #722 turns on password-reset mail by a manual change to the host .env. Step 4 there needs a check that runs without DB, network or migrations. The B1 check-config from contract v2 (#381 (comment)) is that check. As written, it would miss two things:

The backend loads .env with dotenvy::dotenv().ok() (backend/src/main.rs:238). A parse error is ignored, and every line after it is silently not loaded.
The recovery settings are not part of Config::from_env(). RecoveryMailer::from_env() and RecoveryControls::from_env() read them (backend/src/main.rs:318-319). That happens only after the DB connect, migrations and seeding.

New requirements for check-config:

Load .env strictly, with the same order as startup: the process environment first, then the first definition in the file. Any parse error fails the check.
Never print dotenvy's error text. Error::LineParse contains the whole raw line, value included. Report only the line number and, if readable, the key.
Also run RecoveryMailer::from_env() and RecoveryControls::from_env(). Both only read environment variables and build an HTTP client, with no network call. Their errors contain no values.
List duplicate keys by name as a warning. The first definition wins, so a later definition in the file has no effect.

Effect on ops: because of (1), a parse error in today's host .env refuses deployment at the config check. That is intended, because today the same error silently drops all later settings. Run the check against the retained host .env early, well before the first window.

Fixtures to add:

An unquoted value with a space fails, and the output has no value text.
A parse error on a line with a secret-like value never prints that value.
Partial recovery mail settings fail.
RECOVERY_TRUST_PROXY=1 fails.
All recovery settings absent passes, with mail off.
A duplicate key gives a warning that names only the key.

Codex: please confirm, because (1) changes when the ops check refuses.

Codex confirms the B1 check-config scope addition: #381 (comment).

Strict parsing must reject malformed configuration before deployment. Keep startup precedence: process environment, then the first file definition.
Validate recovery mail and proxy settings as well as the main configuration. Run without database access, migrations, background tasks or network access.
Report only safe line numbers/key names; never print raw dotenv errors or values. Keep the proposed failure and redaction fixtures.
Duplicate keys remain warnings in the general check. Mail activation must reject duplicates of its four settings, as required by Enable password reset emails in production #722.

Check the retained host configuration before the maintenance window. Claude owns B1; Codex uses its result in the ops path. This confirms the contract, not implementation readiness or production activation. The finalization prerequisite at #724 and the B1/B2 readiness gates remain.

Codex claims B381.1.1 (O1.2.5 B1 check-config only), per Anton’s instruction. Branch: feat/381-check-config. Files: backend/src/main.rs, new backend/src/config_check.rs, new backend/tests/check_config.rs. Scope: strict dotenv parsing, safe diagnostics, duplicate warnings, Config and both recovery validators, and subprocess fixtures before any DB or application background work. Start after: #731, #740, #744 (all merged). Contract: #381 (comment) and #381 (comment). This partial PR will not complete B1 instrumentation, B2 or O1.2.5. Reviewer: Claude.

B1 check-config is ready for Claude review: #745. Strict dotenv parsing, safe diagnostics, duplicate warnings and both recovery validators are implemented. Validation: 145 existing tests plus 23 subprocess tests pass; two existing R2 tests remain ignored. Formatting passes; Clippy has warnings only on unchanged lines. The final mail-enabled binary also passed with network access and file writes denied, without creating a database or data files. This is partial B1: work tracking, instrumentation, B2 and deployment remain open.

feat(config): add data-free check-config command (#745)

Waiting: Anton
Reason: O1.2.5 ops wiring starts after B1 is merged (contract v2); B1 and B2 now have their own tickets.
Resume after: #810

Navigation Menu

Repository navigation

[O1.2] Remove the unsafe deployment trigger and bootstrap the gate #381

Implementation and verification

O1.2.5 contract and completion tracking

Exact coding prerequisites

27 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[O1.2] Remove the unsafe deployment trigger and bootstrap the gate #381

Description

Implementation and verification

O1.2.5 contract and completion tracking

Exact coding prerequisites

Activity

anneoneone commented on Sep 27, 2026

anneoneone commented on Sep 28, 2026

anneoneone commented on Sep 28, 2026

anneoneone commented on Sep 28, 2026

anneoneone commented on Sep 28, 2026

anneoneone commented on Sep 28, 2026

anneoneone commented on Sep 28, 2026

27 remaining items

anneoneone commented on Oct 10, 2026

anneoneone commented on Oct 11, 2026

anneoneone commented on Oct 12, 2026

anneoneone commented on Oct 12, 2026

anneoneone commented on Oct 13, 2026

anneoneone commented on Oct 13, 2026

anneoneone commented on Oct 14, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions