Skip to content

ci: stop releasing commits whose main CI failed, and confirm each deploy is live - #15

Merged
beastyrabbit merged 3 commits into
mainfrom
ci/publish-requires-green-ci
Oct 22, 2026
Merged

beastyrabbit merged 3 commits into
mainfrom
ci/publish-requires-green-ci

Conversation

@beastyrabbit

Copy link
Copy Markdown
Owner

Note

🤖 Claude Opus 5.5 responding on behalf of beastyrabbit

v0.15.0 shipped from a commit whose main CI had failed. The Verify supported runtimes run on 6def5ab0 failed on Linux (macOS was cancelled), yet the tag publish run for that same commit pushed the container, the npm package and the release assets. The release job only checked that the tag was on main; it never looked at CI, and main has no branch protection to catch it.

Tag releases now wait for main CI on the exact tagged commit and publish nothing unless it passed. After publishing, a new job confirms that schaffa.dev is actually serving the release.

Release gate

Before any publishing step, the publish job runs scripts/ci-require-green-main.mjs. It asks GitHub for verify.yml runs from a push to main on $GITHUB_SHA, checks those fields on every returned run itself, and takes the newest one. It does not filter for successes, so a newer failure always beats an older pass.

  • completed + success: the release continues and the log shows the run URL and attempt.
  • Missing, queued or in progress: it waits up to 10 minutes, because tags often land seconds after the main push.
  • Completed with any other conclusion: it fails right away.
  • API errors, non-JSON, incomplete pagination, malformed runs or unknown statuses: it fails closed.

A pull request run, a run on another branch or a success on a different commit cannot satisfy it. Re-running a failed verify.yml run updates that same run, so a green re-run unblocks the release. The job gains actions: read; the ancestry check stays.

Because the gate guarantees main CI, the publish job no longer repeats pnpm typecheck && pnpm lint && pnpm test or installs ffmpeg (about a minute). Nothing later depended on them: pack:cli builds the CLI itself, the image builds in Docker, and the CLI artifact check does not use ffmpeg. Version validation, the npm duplicate check, gitleaks, the audit, CLI artifact verification, the image scan and digest promotion, npm publish, release attachments and the source map job are unchanged, as are verify.yml, arc-smoke and .forgejo/.

Post-deploy smoke test

/healthz is behind Pangolin SSO publicly (401), so the running release had no public signal. /api/capabilities now also returns serverVersion (for example 0.16.0). The existing version: 1 stays the response schema version. /healthz is unchanged.

A new deploy-smoke job runs after publish on stable tags. Prereleases skip it, because the Flux image policy (>=0.0.0) never selects them. It polls https://schaffa.dev/api/capabilities for up to 20 minutes, which covers the 2-minute image scan, the 2-minute pin commit, the Git sync and the Recreate rollout. Once the expected version is live, it checks that the landing page and capabilities API return 200 with the expected content. A newer version fails as superseded, not as success. Malformed responses, redirects, unhealthy endpoints and timeouts fail loudly. The first release with serverVersion is handled: the current 0.15.0 counts as an older release while the job waits.

Verification

  • Fixture tests with mocked responses, no network. The gate covers an exact-SHA success, a wrong SHA, event, branch or workflow, queued then success, failure, cancelled, missing until timeout, API errors, pagination, incomplete pages, and a newer failure after an older success. The smoke test covers old, expected and newer versions, malformed responses, timeouts and unhealthy endpoints. pnpm test:release passes 26/26.
  • pnpm typecheck, pnpm lint, pnpm build, pnpm security and actionlint pass. CLI tests pass 50/50 and server tests 124/125. The one failure is the local video test: Chrome's sandbox is blocked on that host and ffmpeg is missing.
  • Read-only checks against the real GitHub API: the gate blocks 6def5ab0 (failed run 39666551720) and passes 59518960 (run 39664865751). Against production, the smoke script reports 0.15.0 as a release without serverVersion.
  • Independent review by GPT-6.1 Sol (high effort) found one issue: an incomplete page could hide a newer failure. It's fixed in the second commit, and the follow-up review found nothing.

Note for the next release

Main CI on 6def5ab0 is red because of a timing-sensitive browser test. In test/telemetry-browser.test.ts, "early queued error reaches the local envelope stub" allows about 2.5 s for the GlitchTip envelope. It passed in the publish job and locally on the same commit. With this gate, a flaky failure on the release commit blocks the tag until verify.yml is re-run green, which is the intended behavior.

🤖 Generated with Claude Code

beastyrabbit and others added 2 commits October 22, 2026 15:25
Require the newest verify.yml main push run for the exact tagged SHA to
complete successfully before publishing any release artifacts.

Remove duplicate publish-job type, lint, test, and ffmpeg setup while
retaining the release validation and security checks.

Expose serverVersion through the public capabilities API without changing
its schema version or the health endpoint.

Smoke-test stable deployments after publishing, waiting for the expected
public version and checking the landing page and capabilities endpoint.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reject missing runs and inconsistent totals before selecting the newest run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-22T23:33:02.453844Z 252fb05 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Extract workflow run and endpoint validation, simplify SemVer parsing,
and use non-mutating sorting and top-level await. Add SemVer edge cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@beastyrabbit

Copy link
Copy Markdown
Owner Author

Note

🤖 Claude Opus 5.5 responding on behalf of beastyrabbit

@codex review

New head 252fb05 fixes the six SonarQube quality-gate issues in the two new CI scripts without changing behavior. It extracts the run and endpoint validation into helpers, uses toSorted, uses top-level await, and replaces the long SemVer regex with a short one plus identifier checks in code. It also adds SemVer edge-case tests.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: 252fb05b2b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@sonarqube-skyway-gmbh

Copy link
Copy Markdown

@beastyrabbit
beastyrabbit merged commit 4c81abf into main Oct 22, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant