Repository navigation
Shards precompiling into one shared depot can race, and sharding the whole fleet made it likely #60
Description
A shard died on the default branch with
ERROR: LoadError: The following 1 direct dependency failed to precompile:
SystemError: opening file "…/.julia/compiled/v1.12/Markdown/AREjX_vfzSw.ji": No such file or directory
One shard read a cache file while another was replacing it. On a self-hosted pool the depot is persistent and shared per box, so N shards on the same box precompile into the same ~/.julia/compiled.
Why this is worth writing down rather than re-running past
The probability is a function of how many jobs precompile concurrently, and that number just changed by an order of magnitude. Before today, one repository ran one test job. Today thirty-odd repositories each run four to sixteen shards, on the same pool, and they will keep doing that on every push. The migration did not introduce the hazard; it moved it from "essentially never" to "expect it".
The failure is at least loud, and a re-run clears it. But a fleet where any run can fail for a reason unrelated to the code is a fleet where red checks stop being read — which is the thing several of this package's other guards exist to prevent.
What might fix it
julia-actions/cachealready handles the hosted case; the self-hosted case is the one with a shared writable depot.- A per-shard depot —
JULIA_DEPOT_PATHunder the runner's workspace, with the shared one appended read-only. Costs precompilation time per shard, which is exactly what the shared depot was buying. prebuildalready exists and sidesteps it by precompiling once — but it is documented hosted-only, because Julia keys a package's compiled cache on its absolute source path and that path carries the runner's name on a self-hosted pool. Worth re-checking whether that reasoning still holds for DEPENDENCIES, which is where the race is: the failing file wasMarkdown, a stdlib, not the package under test.
That last point may be the cheap fix: dependencies do not change per commit and their caches could be built once and shared read-only, leaving only the package under test to be compiled per shard.
Status
Two repositories hit it on the same afternoon; both re-runs are pending as I write this. Filing before knowing whether they clear, because "the re-run was green" is not evidence the race is gone — it is evidence that a race is a race.
Activity
sotashimozono commentedon Aug 3, 2026
Both re-runs are green, which is what a race looks like and not a reason to close this.
ITensorcMPS.jlcompiled/v1.12/Markdown/….ji: No such file or directory. Green on re-run. Cause known, transience confirmed.SingleSiteDMRG.jlSo: one confirmed instance, one that merely resembles it.
The reason to keep this open is not the two failures. It is that both landed within an hour of each other, on the afternoon the pool went from running one test job per repository to running four-to-sixteen shards per repository across thirty of them. That ratio is now permanent. Two in one afternoon at the very start of the new steady state is the number worth acting on, not the fact that re-running cleared them.
The narrow observation from the confirmed case still looks like the cheap fix: the file that lost the race was
Markdown, a stdlib — a dependency, not the package under test. Dependencies do not change per commit, so their caches could be built once and shared read-only, leaving only the package under test compiled per shard. That isprebuild's idea applied to the half of the problem where the absolute-source-path objection does not bite.sotashimozono commentedon Oct 14, 2026
Hermes の整理（自動。何も変更していません）
状態: 提出から変更なし。main の
sharded-tests.ymlに、race を避ける変更 (shard ごとの depot、prebuildの self-hosted 対応) は見当たりません。コメントに「確認できた発生は 1 件 (ITensorcMPS.jl)、もう 1 件 (SingleSiteDMRG.jl) は log 失効で未確認」とあり、その後の再発は要確認です。論点: 本文の「依存だけ先に共有 read-only で作れば足りるのでは」は、#14 の 2026-07-29 の計測 (依存は
$HOME/.julia/packagesから compile されるので job をまたいで使える。壊れるのは$GITHUB_WORKSPACE由来の被試験 package だけ) と矛盾しません。ただしprebuildの description は今も「HOSTED RUNNERS ONLY」で、self-hosted 向けに依存分だけ使う経路は未実装です。関連: #14 (依存は bridge する、の計測), #72 (同じ depot を N shard が保存する問題), #62 / #71 (self-hosted では cache を切る)。注意: #71 の
cache: falseは restore/save を止めるだけで、共有~/.julia上の precompile 競合そのものは減りません (要確認)。