Skip to content

[CI/Packaging] Publish each PR staging snapshot with one finalizer - #1089

Closed
brandonpayton wants to merge 4 commits into
mainfrom
ci/staging-single-writer-index-qk044
Closed

[CI/Packaging] Publish each PR staging snapshot with one finalizer#1089
brandonpayton wants to merge 4 commits into
mainfrom
ci/staging-single-writer-index-qk044

Conversation

@brandonpayton

@brandonpayton brandonpayton commented Jul 24, 2026

Copy link
Copy Markdown
Member

Why

PR package staging currently makes every package worker take the same release
lock, download release state, upload one archive, and replace index.toml.
That turns a large parallel package wave back into a mostly serial publication
queue. It also lets an interrupted first run leave a sparse release that later
runs cannot safely reuse.

This change keeps package compilation parallel and gives release mutation to
one auditable writer after the full build wave is known. Package identity,
dependency ordering, last-green behavior, and consumer testing remain
fail-closed.

What this changes

  • Library and program workers have read-only repository access. They build in
    dependency order and upload immutable workflow artifacts, with up to ten
    workers running at once.
  • One post-matrix finalizer freezes only the exact-current usable subset of
    canonical state, trims it to the current managed package set, and applies
    every successful matrix artifact offline. Missing, stale, transient, and
    failed-without-fallback keys become matrix-owned gaps. An exact-current
    failed entry may contribute its fully verified last-green fallback.
    Final validation still requires exact full coverage.
  • When no canonical release exists for a new ABI, the matrix must provide the
    complete first snapshot. Missing or invalid artifacts are recorded as
    truthful failed entries; the finalizer never invents a fallback.
  • Before publication, the finalizer requires exact package/architecture
    coverage, version, revision, cache key, immutable Git provenance, safe
    target-relative URLs, and archive manifest/hash/size agreement.
  • The final asset plan contains only archives referenced by the completed
    index. Disabled, historical, and unrelated canonical assets are not copied.
  • The sole writer resolves the release ID once, reads every asset page through
    retry-safe stable snapshots, rejects duplicate or changing inventories,
    uploads or verifies immutable archives, replaces the complete index as the
    final mutable write, and re-reads all referenced bytes.
  • A failed package may still let consumer tests run only when its last-green
    fallback has the same version, revision, cache key, and immutable Git
    provenance. A separate package result remains red, so testing a fallback
    never hides the failed current build.
  • Status reporting distinguishes matrix build results, release publication,
    and consumer test results.

GitHub implements gh release upload --clobber index.toml as delete followed
by upload, not an availability-atomic swap. Publishing archives before the
index keeps every visible index referentially safe, but interruption can leave
the index temporarily absent until a retry repairs it. Canonical activation's
journaled publisher is intentionally unchanged.

Validation

  • bash scripts/dev-shell.sh env AR=llvm-ar bash tests/scripts/package-publish-flow.sh
  • bash scripts/dev-shell.sh cargo test -p xtask --target aarch64-apple-darwin staging_reuse --no-fail-fast — 31 passed
  • /opt/homebrew/bin/actionlint -color .github/workflows/staging-build.yml
  • ShellCheck on every changed/new shell script (SC2034 excluded only for the
    pre-existing unused variable in test-merge-candidate-workflows.sh)
  • git diff --check

The local tests include package addition and architecture addition against an
existing partial baseline; repair of canonical failed, stale, transient, and
missing keys; exact failed-fallback materialization and later success cleanup;
partial or malformed fallback rejection; a library-to-program package-kind
change; complete and failed first-ABI composition; an index-only all-failed
publication; a later-page asset on a release with more than 100 assets;
partial-response retry; idempotent retry; duplicate metadata; conflicting
immutable bytes; and the workflow outcome truth table.

Hosted proof and current external gate

This PR is stacked on #1085's frozen head so it can be reviewed independently
from the active product rollout. Its own CI-control-only diff does not select a
package matrix, and the canonical binaries-abi-v42 release does not exist
yet, so its current hosted test preparation fails before exercising the new
finalizer.

After this exact head passes independent review, a separate
lsof package-input canary will be rebased onto it. That canary is
never-merge test scaffolding: it intentionally triggers the complete
no-baseline ABI 42 matrix and this single-writer finalizer.

Keep matrix builders read-only and collect their immutable archives before any release mutation. Validate a complete target-relative snapshot offline, acquire the staging lock once, publish the index only after every referenced archive is present, and re-read the published transaction. Preserve last-green failure evidence and distinguish build success from publication success.
Let the test gate consume only exact-current last-green fallbacks after the single writer has published and re-read a complete snapshot, while a separate package result remains red for any failed build.

Resolve the release ID once, read every asset page through retry-safe stable snapshots, reject duplicate or changing inventories, and cover first-ABI, page-two, partial-response, conflict, and index-only publication paths.
Freeze every valid expected package/architecture key available in canonical state without requiring absent keys at the baseline-only step. Matrix artifacts can then add or repair those keys, while the post-composition validator still requires exact full coverage.

Cover package additions, architecture additions, partial archive materialization, unexpected entries, and falsely complete snapshots.
Base automatically changed from homebrew/formula-source-reference-contract-qk044 to main July 24, 2026 21:16
@brandonpayton

Copy link
Copy Markdown
Member Author

Superseded by merged #1160. The current design replaces this draft
mutable staging finalizer with exact release identities, sparse ledgers,
and immutable publication. Closing the older alternative so future
work follows the landed contract.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant