[Homebrew/Test] Isolate Brew tap memory across browsers - #1168
Closed
brandonpayton wants to merge 28 commits into
Closed
[Homebrew/Test] Isolate Brew tap memory across browsers#1168brandonpayton wants to merge 28 commits into
brandonpayton wants to merge 28 commits into
Conversation
Every push to main starts a Pages workflow. Cancellation is only a resource optimization; it does not decide which run may publish. Document that the deployment gate admits only the newest run triggered for main, so a delayed older run cannot replace a newer site.
The fail-closed development shell drops ambient workflow variables. The Pages acceptance step set its product identities only on the outer step, so the browser could silently receive defaults instead of the admitted shell contract. Pass every selected identity through the explicit dev-shell boundary. Extend the deployment checker and its mutation suite so deleting any required handoff fails before Pages can publish.
The shipping gate now proves first- and third-party installs in two fresh Node processes while Chromium retains the complete lifecycle. State that exact contract in the Pages cutover docs.
Vite imports a content-addressed shell image under dist/assets. The un-hashed public copy can be stale while the browser loads different bytes. Make Pages validate the asset Chromium will use, and lock that choice into the deployment contract with positive and negative checks.
Run first-party and independent-tap shipping proofs as separate browser scopes. Keep the comprehensive reboot lifecycle as a separate soak. Fail immediately when Chromium loses its renderer, stream only bounded guest progress, and collect bounded cgroup and per-Chromium RSS telemetry. This turns an early page crash into useful evidence instead of an eighteen-minute timeout.
Bind the immutable mirror release and exact plan in the recovery lock. Let public fixtures omit local bottle payloads because Chromium verifies the plan-owned URLs, sizes, and hashes as it downloads each lazy layer.
Capture each fresh runner cgroup OOM baseline and require it to remain unchanged. Preserve the counters beside browser and process telemetry so a renderer loss cannot be mistaken for a successful bottle install.
Build the sealed browser once, then give the core and canary scopes separate fresh Chromium runners. Fetch immutable inputs anonymously, bind the public mirror plan, fail on page loss or cgroup OOM, and retain bounded process telemetry. Keep product and runtime-source authorities distinct. Static mutation tests prevent proof consumers from regaining Nix, Vite, resolver, or publication work.
A consume-only browser proof must not repeat the product build on the memory-constrained runner or confuse test-code provenance with the published product authority. Create one deterministic handoff containing the sealed browser dist, the exact public fixture, and only the Playwright source needed to run it. Record the immutable product commit separately from the runtime source commit. Reject traversal, symlinks, special nodes, inventory drift, changed bytes or modes, and any relaxed resource bound. Use a minimal pinned Playwright package and a fixed static server so the consumer installs npm and Chromium but does not run Nix, Vite build, kernel fetch, or publication. Validation: - browser runtime handoff contract suite - shell syntax and ShellCheck - Node syntax for the static server - minimal npm install and Playwright shipping-test discovery The live public Chromium lifecycle was not run by this commit.
Remove GitHub Actions credentials before Playwright starts Chromium and the sealed static server. This keeps the public bottle proof anonymous even if later transport code begins reading its parent environment.
Run the browser runtime handoff mutation suite from the aggregate Homebrew publisher contract so inventory, authority, mode, and resource bounds cannot drift outside focused development.
Require the recovery lock to retain the exact lifecycle release asset sizes and SHA-256 values. A syntactically valid replacement digest must not silently redefine the immutable product being proved.
A future edit to the proxy server, fixture contract, or handoff verifier must not bypass the live public Chromium proof merely because the workflow path filter did not name that executable input.
The consume-only Chromium proof serves sealed product bytes without Vite. The current browser still sends public bottle, OCI, and Git traffic through its same-origin CORS proxy route. Omitting that route makes the sealed proof fail before it reaches Kandelo. Add a temporary, bounded transport seam to the sealed server. Permit only GitHub and GHCR public hosts, pin public DNS results, stream bounded responses, and support the GET, HEAD, and POST behavior required by Homebrew and Git smart HTTP. Relay only guest-created protocol headers, remove authorization on redirects, and preserve safe range selectors across GitHub CDN redirects. Document service-worker ownership as the intended architecture. Cover GHCR, encoded Git POSTs, redirects, credential stripping, range headers, null-body statuses, SSRF rejection, size limits, timeouts, and upstream failures. Validation: - canonical browser runtime handoff contract suite - ShellCheck and Node syntax checks - minimal npm install and Playwright shipping-test discovery - live public release GET and Range reads through production policy - live public Git smart-HTTP advertisement through production policy The live public Chromium lifecycle was not run by this commit.
Keep the public Chromium recovery beside the product cutover it admits. This avoids a second main merge and gives the live proof the exact combined PR merge as its runtime authority.
The sealed-handoff verifier executes Node, but the producer called it through the runner ambient PATH after other tool setup. That PATH no longer selected Node 24 in the live proof. Run handoff creation inside the repository dev shell, lock that boundary in the workflow checker, and report the observed major on future version mismatches.
The Chromium handoff must seal an exact, clean source checkout. The normal musl build applies Kandelo overlays inside its submodule, so an in-tree build makes a valid handoff look contaminated. Archive the exact synthetic-merge tree and musl gitlink into runner temporary storage, build both browser sysroots there, and copy only the generated outputs back. Keep the full clean-tree check and include a bounded path preview when it fails. Validation: - handoff and workflow mutation tests - real isolated wasm32 and wasm64 musl builds - Ruby syntax, shellcheck, actionlint, and git diff checks
The sealed browser manifest records every file, including the Vite Doom asset .gitignore. Artifact upload excludes dotfiles by default, so both fresh consumers received 103 of 104 signed members and rejected the incomplete inventory before Chromium could start. Enable hidden-member transport only for the bounded handoff upload. Require that setting in the canonical workflow checker and its mutation test. The manifest verifier still authenticates every permitted path, size, digest, and mode, so unexpected files fail closed. Validation: - handoff and workflow mutation tests - Ruby syntax, shellcheck, actionlint, and git diff checks - downloaded run 30618124505 artifact inventory comparison
Repeated spawn and exec of the same VFS executable compiled another WebAssembly.Module every time. Short-lived Homebrew commands could therefore leave many compiled-code allocations awaiting engine collection after their process memory had retired. Key one bounded, cross-host LRU by the complete prepared byte snapshot length and SHA-256. This lets aliases share compilation while a write or replacement always selects new code. Coalesce concurrent misses, compile oversized modules uncached, and preserve the existing malformed-initial-image trap path. Cover content aliases, same-length replacement, concurrent reuse, eviction, failed compilation, and a real Node VFS replacement.
Precompiling initial executables in the kernel Worker must not change how a malformed initial module fails. A CompileError still belongs to the process-Worker loader path, while exec and posix_spawn reject bad modules during preflight. Launch a header-valid but truncated module through the real Node host. Verify that it creates the process, exits through the loader-error path, and reports the existing host diagnostic.
Keep the new cache and its focused unit tests consistent with the host TypeScript formatting contract.
The Lamp, Nginx/PHP, and WordPress image builders boot Kandelo while generating or prewarming data. Their declared host/src input changed with executable-module caching, so regenerate the authoritative program projection instead of admitting stale cache keys.
Build only the focused Homebrew fixture for this do-not-merge measurement. The complete product build now needs three unrelated image regenerations because host/src is one of their declared inputs. Keep the exact immutable shell, mirror, core, and canary inputs. Final #1147 must still prove the complete main product after those images are rebuilt.
The diagnostic build deliberately excludes unrelated generated images, but Vite still imports the current packaged kernel. Resolve only that package through the declared binary cache and bind the same cache to the build. Keep the workflow checker exact and add a mutation test that rejects substituting another package.
Even the focused Vite graph imports the kernel and generic rootfs. Resolve only those two packaged inputs through one isolated cache. Keep unrelated generated browser images outside this diagnostic. Extend the exact workflow checker and its mutation test accordingly.
Record bounded cache hit, miss, and eviction counters before a renderer crash. Preserve Playwright browser-process diagnostics so the next throwaway proof can distinguish cache thrash from a Chromium failure.
Regenerate the three host-backed image identities after adding bounded Chromium cache telemetry. This keeps focused binary resolution fail-closed without building unrelated demo images.
brandonpayton
force-pushed
the
test/executable-module-cache-public-chromium-qk044
branch
from
July 31, 2026 13:46
8824329 to
2c2b900
Compare
Contributor
Phase B-1 matrix build status —
|
| Package | Arch | Status | Sha |
|---|---|---|---|
| lamp | wasm32 | failed | — |
| nginx-php-vfs | wasm32 | failed | — |
| wordpress | wasm32 | failed | — |
Auto-generated; replaced on each push. Raw data in the publish-status workflow artifact.
brandonpayton
changed the title
Member
Author
|
Closing this do-not-merge diagnostic after the bounded Pages cutover.\n\nThe module-cache experiment did not prevent the Chromium renderer\ncrash. Product cache work remains in #1167; the failed fresh-runner\nproof stays here as diagnostic evidence. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Fresh Chromium core and canary runs both approached the hosted
runner's 16 GB memory ceiling while stock Brew tapped a repository.
The renderer then crashed.
PR #1167 found that Kandelo recompiles the same WebAssembly executable
for every short-lived process. It adds a bounded compiled-module cache.
Focused tests cannot show whether that change controls a real Brew
workload in Chromium.
This diagnostic branch combines the public Chromium lifecycle with
#1167. It exists only to run the exact immutable core and canary inputs
on fresh hosted runners.
Hosted Chromium result
Run
30635830180disproved cache thrashing as the complete cause.Both independent runners reached the same cache state:
The core runner still peaked at 13,915,262,976 cgroup bytes. Its main
renderer later held about 6.2 GiB. The canary runner peaked at
14,345,535,488 cgroup bytes and later held about 9.4 GiB in its main
renderer. Both renderers crashed with zero cgroup OOM and OOM-kill
events.
Both failures happened during the common first-party
brew tapphase,before canary-specific work. Neither run reported a guest trap, a fork
continuation error, a Ruby exception, or an incorrect child status.
Matched cross-browser result
The same sealed core fixture was then run with Playwright's Chromium,
Firefox, and WebKit engines on one Mac. The fixture and its 38 inputs
were verified by digest before each run.
brew tapRSS means resident set size: the physical-memory pages attributed to
the browser process tree at that sample.
Firefox completed in 5.7 minutes, and its RSS fell between allocation
waves. Chromium reproduced the hosted failure in 61 seconds on the same
machine. Its main renderer alone reached about 6.43 GiB.
WebKit did not reproduce the Chromium memory failure. It failed earlier
inside instrumented Ruby with
Maximum call stack size exceeded. Thestack was Ruby Wasm compilation and
requirework, after the module hadalready instantiated. That matches the known WebKit stack limit for an
overly broad
wasm-fork-instrumentclosure.These results rule against a simple cross-engine Kandelo memory leak.
They do reveal an architectural workload amplifier: the older Ruby
creates hundreds of process workers and fresh Wasm memories. Chromium
retains or collects that churn differently enough for its renderer to
crash. Kandelo should remove avoidable churn rather than rely on one
engine's garbage-collection timing.
Current interpretation
The immutable fixture uses the older fork-heavy Ruby bottle, not
#1166's direct-
posix_spawnRuby. The Chromium trace contains 712process-worker script requests but only 160 executable-cache requests.
The extra worker generations therefore reuse executables while still
creating fresh process memories.
The old Ruby imports a minimum of 205 Wasm pages, or 12.8125 MiB.
Multiplying that minimum by roughly 552 extra worker generations gives
about 7 GiB of cumulative backing churn. That is close to Chromium's
observed peak, but it is an inference; it does not prove that all those
memories were live at once.
There is no observed semantic
wasm-fork-instrumentfailure in theChromium trace. Such a failure would normally produce a continuation
trap, duplicate effects, an incorrect child status, or another guest
error. WebKit's separate stack failure still requires focused
instrumenter follow-up.
Diagnostic boundary
Changing
host/srcalso changes three unrelated generated imageidentities: Lamp, Nginx/PHP, and WordPress. The final product must
regenerate those images before #1147 can build and prove the complete
main site.
This do-not-merge measurement builds only the focused Homebrew fixture.
It still uses the exact public shell, bottle mirror, core tap, and
independent canary tap. Its failure is diagnostic evidence; it is not
the final main-product proof required to merge #1147.
Next discriminator
Keep the fixture, host, cache, kernel, rootfs, and tap revisions fixed.
Replace only Ruby with #1166 revision 13, then record fork, spawn, exec,
exit, worker-launch, and memory counts in all three engines.
If fork counts and memory collapse, the old Ruby process path caused
the remaining pressure. If forks are nearly absent but memory remains,
the host launch and executable-snapshot path is responsible. A
low-memory residual fork that traps or returns a wrong status would
instead implicate fork instrumentation.
Do not merge this pull request. Land product corrections through their
separate reviewed pull requests, then close this diagnostic branch.
Scope
The product inputs are unchanged from #1147. The runtime difference is
#1167's bounded compiled-executable cache. The diagnostic workflow also
removes unrelated demo images from this measurement only.
This does not change the kernel, guest ABI, process-memory ownership,
packages, bottles, or VFS format.