[Packaging/CI] Make shell source fallback builds self-contained

[brandonpayton]

Why

A package archive can exist but still be rejected because its checksum, architecture, ABI, or cache identity is stale. The package resolver then correctly falls back to building that package from source.

For example, a lamp build can pull in the shell package indirectly. If the staged shell archive is stale, the resolver starts the shell source recipe. Previously, only CI jobs that selected shell directly installed the JavaScript tools used to compose its VFS image, so the indirect fallback failed even though the resolver was doing the right thing. Local callers and overlapping builds had the same hidden dependency on shared checkout state.

A source recipe must work from every direct or transitive resolver caller. It must not depend on a caller guessing that fallback will happen, and concurrent builds must not replace one another’s node_modules trees.

What changed

• The shell recipe now creates a private source snapshot below its resolver-owned output directory. It installs the root and mkrootfs lockfiles there and runs the VFS composer only from that snapshot.
• Every Git command used to create the snapshot runs through one isolated wrapper. Ambient Git attributes, replacement refs, repository selectors, external diff configuration, credentials, and user configuration cannot silently change package inputs. Untracked repository info/attributes fail clearly before npm runs.
• npm starts from an empty environment, uses private HOME/config/cache directories, and is pinned to the public npm registry. GitHub/npm credentials and NODE_OPTIONS/NODE_PATH cannot reach the installer or composer.
• run.sh and package-publishing workflows no longer predict whether the resolver will source-build shell or mutate checkout-global dependencies.
• The new preparer is part of build.toml.inputs; npm and tar are declared host tools. The package revision is unchanged because the intended VFS bytes did not change, while the changed build inputs still produce a new cache identity.
• The authoritative program-packages.json projection is regenerated so shell and every composite package that depends on shell carry the new manifest and cache identities.
• Tests cover stale same-run overlays, current tracked bytes, hostile Git/npm configuration, symlink substitution, missing installer outputs, concurrent private snapshots, and cleanup.

Validation

Passed on corrected head 91be65e4236714e6a6ac830a54f5d89b0230840b:

• bash scripts/dev-shell.sh bash scripts/test-package-build-roots.sh
• Exact browser package-root discovery used by the lazy-shell job: browser-binary-package-roots.mjs --exclude-package shell --include-package rootfs
• bash scripts/dev-shell.sh bash scripts/test-homebrew-main-shell-closure.sh
• bash scripts/dev-shell.sh bash packages/registry/shell/test-prepare-build-tools.sh
• bash scripts/dev-shell.sh bash scripts/test-homebrew-publish-workflow.sh (run before the generated projection-only correction; all implementation inputs are unchanged)
• AR=llvm-ar cargo test -p xtask --target <host> stale_direct_pr_overlay_falls_through_to_source_build -- --nocapture (1 passed)
• cargo run -p xtask --target <host> -- build-deps parse shell
• Shell syntax checks and ShellCheck for the new/changed shell package scripts
• Actionlint for the three changed workflows, excluding the existing unrelated SC2006 warning at force-rebuild.yml:113
• git diff --check

The final declared-tool-path correction on faef5ead8 also passed the shell tool-preparer test, the complete Homebrew main-shell closure contract (17 assertions plus two concurrent real npm installs), the package build-root suite, Bash syntax validation, and git diff --check.

The aggregate publisher suite includes publisher trust, immutable VFS release recovery, 17 shell image-contract cases, and two concurrent shell source-fallback builds.

The first CI head correctly reported that program-packages.json was stale after the shell manifest/cache inputs changed. Both failing jobs had this same cause; it was a PR freshness defect, not an ABI 42 or tap-catalog blocker. The corrected head contains the generator-produced projection and passes both failing commands locally.

Scope and rollout

This is a packaging/build fix. It does not change the kernel, ABI, host runtime, browser runtime, formula catalog, or intended VFS contents.

This packaging-only PR is rebased on current main after #1091 and can merge independently of the frozen #1079 ABI 42 publication lineage. Future publication pins still must use immutable commits from the merged ancestry.

Residual boundary: integrity-pinned npm lifecycle scripts still execute with ordinary operating-system permissions for the runner user, although their credentials, configuration, and build paths are isolated.

[github-actions]

Phase B-1 matrix build status — pr-1086-staging

ABI v41. 6 built, 0 failed, 6 total.

Package	Arch	Status	Sha
shell	wasm32	built	8491816d
lamp	wasm32	built	80ed352c
nginx-php-vfs	wasm32	built	2a73a387
nginx-vfs	wasm32	built	af83f43e
node-vfs	wasm32	built	f9b06103
wordpress	wasm32	built	d98078b1

_{Auto-generated; replaced on each push. Raw data in the publish-status workflow artifact.}

[brandonpayton]

Closing as superseded by #1109 commit ee45cb47e, which carries the self-contained shell fallback on merged #1108 authority. The replacement retains the stale-overlay source-fallback regression and does not use this branch’s old predictive workflow setup.