Load FileWatching before precompilepkgs in ensurecompiled - #1135
Merged
Merged
Conversation
Fixes #1134. Since #1131, `ensurecompiled` runs `Base.Precompilation.precompilepkgs()` under the base sysimage. On Julia 1.11 `precompile_pkgs_maybe_cachefile_lock` does `Base.loaded_modules[FileWatching]`, which assumes `FileWatching` is already loaded. That holds for the default sysimage but not for the stdlib-free base sysimage of `incremental=false` builds, so every precompile task died with a `KeyError` that is swallowed inside the `@async` task, and `precompilepkgs()` exited successfully having compiled nothing. The `--output-o` step then loaded every package from source. Since Julia 1.11 `jl_atexit_hook` runs `Base._atexit` before `jl_write_compiler_output`, so every sysimage carries `_atexit_hooks_finished = true`, and a nonincremental `--output-o` process skips the `Base.__init__` that resets it. Packages whose top-level code calls `atexit`, like `Downloads` (its precompile block calls `Curl.__init__()`), therefore fail with `cannot register new atexit hook; already exiting.` when loaded from source there. Loading from a cache file avoids the top-level code. `Pkg.precompile()` did not hit this because loading `Pkg` loads `FileWatching`. Julia 1.12+ use `Base.trymkpidlock_hook` with a fallback, so they are unaffected; loading `FileWatching` there is harmless. Adds a nonincremental `create_sysimage(["Downloads"])` test without a precompile execution file, which is the only way the cache files can come from `ensurecompiled`. It fails with the error from the issue without the fix. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests.
Additional details and impacted files@@ Coverage Diff @@
## master #1135 +/- ##
==========================================
- Coverage 83.93% 74.19% -9.75%
==========================================
Files 3 3
Lines 909 903 -6
==========================================
- Hits 763 670 -93
- Misses 146 233 +87 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1134.
Since #1131,
ensurecompiledrunsBase.Precompilation.precompilepkgs()under the base sysimage. On Julia 1.11precompile_pkgs_maybe_cachefile_lockdoesBase.loaded_modules[FileWatching], which assumesFileWatchingis already loaded. That holds for the default sysimage but not for the stdlib-free base sysimage ofincremental=falsebuilds, so every precompile task died with aKeyErrorthat is swallowed inside the@asynctask, andprecompilepkgs()printedPrecompiling project...and exited 0 having compiled nothing. This is visible in the libtrixi CI logs: v2.4.1 precompiled 273 packages in that step, v2.4.3 spent 3 seconds and compiled none.The
--output-ostep then loaded every package from source. Since Julia 1.11jl_atexit_hookrunsBase._atexitbeforejl_write_compiler_output, so every sysimage carries_atexit_hooks_finished = true, and a nonincremental--output-oprocess skips theBase.__init__that resets it. Packages whose top-level code callsatexit, likeDownloads(its precompile block callsCurl.__init__()), therefore fail withcannot register new atexit hook; already exiting.when loaded from source there. Loading from a cache file avoids running that top-level code.Pkg.precompile()did not hit this because loadingPkgloadsFileWatching. Julia 1.12+ useBase.trymkpidlock_hookwith areturn f()fallback when the hooks are undefined, so they are unaffected; loadingFileWatchingthere is harmless.The new test builds a nonincremental sysimage with
Downloadsand no precompile execution file, which is the only configuration where the cache files can come fromensurecompiledalone (the existing app/library tests all run a precompile script under the base sysimage, which compiles the caches through regular loading). Verified locally on Julia 1.11.9 with a fresh depot: it fails with the error from the issue without the fix and passes with it.🤖 Generated with Claude Code