Repository navigation
Dependency load order in StatsFuns 2.x exhausts static TLS on old glibc (libgomp fails to load) #218
Description
Upgrading StatsFuns 1.5.2 → 2.2.0 broke our deployment on an embedded aarch64 platform running glibc 2.27 (Ubuntu 18.04), with:
ERROR: LoadError: InitError: could not load library "libgomp.so.1"
.../lib/julia/libgomp.so.1: cannot allocate memory in static TLS block
during initialization of module CompilerSupportLibraries_jll
This is not a binary regression — the resolved JLL versions (Rmath_jll 0.5.1+0, OpenSpecFun_jll, CompilerSupportLibraries_jll 1.1.1+0, etc.) are identical between the working and failing environments. It's purely a change in the order in which StatsFuns loads its dependencies:
- With
StatsFuns 1.5.x,SpecialFunctions(and henceCompilerSupportLibraries_jll/libgomp) is loaded beforeRmath, and everything works. - With
StatsFuns 2.2.0,Rmathis loaded first.libRmath-julia.socarries an 856-byte TLS segment which, onpre-2.31 glibc(where the static TLS surplus is small, ~1664 bytes, and not tunable), consumes the surplus. WhenCompilerSupportLibraries_jllthen dlopenslibgomp— which requires static TLS (initial-exec model) — the allocation fails.
Minimal demonstration on the affected platform (Julia 1.11.3, glibc 2.27):
using CompilerSupportLibraries_jll, Rmath # works
using Rmath, CompilerSupportLibraries_jll # fails: libgomp cannot allocate static TLS
and consequently using StatsFuns fails at 2.2.0 but works at 1.5.2 with the same artifact versions.
Would you be open to reordering the using statements in StatsFuns.jl so that SpecialFunctions (or CompilerSupportLibraries_jll) is loaded before Rmath? It's a zero-cost change on modern systems and restores compatibility with older glibc, where libraries needing static TLS must be loaded before the surplus is consumed. We're currently pinning StatsFuns to ~1.5 as a workaround.
Activity
devmotion commentedon Jul 27, 2026
Are you using the Julia binaries from julialang.org? Or a Ubuntu package?
mchitre commentedon Jul 27, 2026
From
julialang.org. And its only a problem on Ubuntu 18, not on newer versions.devmotion commentedon Jul 27, 2026
To me this sounds like an upstream problem with libRmath (it seems?) and not something that can be reliably fixed in StatsFuns.
mchitre commentedon Jul 27, 2026
Fair enough, but it is still a problem for users of this library, since the dependency on
libRMathcomes from here.devmotion commentedon Jul 27, 2026
This should be fixed, I just think it should be fixed in one of the upstream dependencies as relying on the load order in StatsFuns is brittle and seemingly users can provoke the issue without StatsFuns by loading these packages manually.
mchitre commentedon Jul 27, 2026
JuliaStats/Rmath-julia#56
mchitre commentedon Jul 27, 2026
Technically this issue was bumped upstream and not fixed, since the issue still exists for users of this library on older Linux systems.
The issue should have been closed as "not planned" rather than "completed", which suggests it's fixed and is misleading. I leave this comment for completeness, and won't follow up on this any further.
Nevertheless, thanks for your quick responses @devmotion, I appreciate it.
nalimilan commentedon Jul 29, 2026
This appears to be a bug in older glibc which is hard or impossible to fix in Rmath-julia (see JuliaStats/Rmath-julia#56 (comment)). If changing the order in which we load dependencies in StatsFuns works around the problem, it seems work doing IMO.
devmotion commentedon Jul 29, 2026
The question is: Does the order of import statements actually guarantee any specific load order? Or is the compiler free to rearrange load order of packages that do not depend on each other? The general approach of this possible workaround seems still too brittle for me to serve as an actual fix for the problem. This is an upstream issue, and without an upstream fix it will still be broken outside of StatsFuns, even if we are able to do something here. Moreover, AFAIU if a user would use
using Rmath, StatsFunsshe would run into the same problem, so it's not even a fix for people using only Rmath and StatsFuns.nalimilan commentedon Jul 29, 2026
There's no way to fix this cleanly AFAICT, so any workaround would be just that. Given that it's very easy to do and it doesn't have any drawbacks it seems worth doing. Very few (if any) people use Rmath directly.
devmotion commentedon Jul 29, 2026
So far nobody has confirmed that this is a stable workaround at all.
Moreover, I don't understand the underlying problem: why does the load order matter if both packages need to allocate TLS segments? Shouldn't the static surplus be exceeded regardless? And as the pytorch issue implies this can occur with basically any library that exceeds the surplus, so loading other packages with TLS-dependent binaries could provoke the same issue? If that's the case, isn't the proper answer just to tell people to use a newer glibc (which AFAIU fixes these issues at the core?)?
devmotion commentedon Jul 29, 2026
I assume people rather use Rmath than Rmath_jll. If the combination of CompilerSupportLibrabries_jll and Rmath_jll is what usually triggers this problem in the Julia ecosystem, another option would be to add a
import CompilerSupportLibraries_jllto Rmath.jl before loading Rmath_jll, to avoid this problem regardless of whether Rmath is used with or without StatsFuns.devmotion commentedon Jul 29, 2026
I looked into this in a bit more detail, mainly to answer my own question above about why the load order matters at all. I used Claude for brainstorming and for working through the details.
Why the order matters: the two libraries are not symmetric.
libRmath-julia.so's TLS is global-dynamic, so it does not have to live in the static surplus at all, glibc < 2.32 just puts it there opportunistically because it fits, as an access-time optimization.libgompon the other hand uses initial-exec, which can only be served from the surplus. So whoeverdlopens first wins, and Rmath winning wastes a resource that only libgomp actually needs. As a side note, this also means that rebuilding with-ftls-model=global-dynamicwould not help, in case someone suggests it: pre-2.32 glibc allocates from the surplus for anydlopened module with aPT_TLSsegment regardless of the TLS model, only the size of the segment matters.Where the change came from: e0a0cab (#188). Removing the
RFunctionssubmodule replacedinclude("rmath.jl"), which sat at the bottom of the module body afterusing SpecialFunctions, with ausing Rmath: Rmathat the very top. So the reordering was an accidental side effect of an unrelated refactoring, and the load order is currently an undocumented invariant that the next refactoring can break again just as silently.A few options that have not been mentioned yet:
Rebuild Rmath_jll with
dont_dlopen=true. JLLWrappers already supports this (@init_library_productskips thedlopenifdlopen_flags === nothing, andlibRmathis still bound to the full path), soccall((:qgamma, libRmath), ...)would resolve the library lazily on the first call instead of atusing Rmath. That takes libRmath out of the load phase entirely, so unlike reordering the imports here it would also fixusing Rmath, StatsFuns. I am not convinced this is worth it though: it only changes when the surplus is consumed, not whether it can be exhausted, so a first Rmath call before CSL is loaded would still fail. On top of that a missing or broken artifact would only error at the first call instead of at load time, and it presumably interacts with static compilation. Mentioning it mainly for completeness.Shrink the TLS segment upstream. Looking at
patches/thread-local.patch, the 856 bytes are mostlyrpois.c(pp[36]plus scalars, ~400 bytes) and the caches inrhyper.c,rbinom.c,rgamma.candrbeta.c. Two things stand out there. First,p1,p2andq1..q4intoms708.care initialized constants that are never written, they should bestatic constand not_Thread_localat all, that looks like a plain mistake in Improve thread-safety Rmath-julia#50. Second, all of these are just caches across repeated calls, so they could be moved behind a single_Thread_localpointer to a lazily allocated per-thread struct, which would bring the segment from 856 bytes down to 8 while keeping the thread-safety of enable travis on 0.7 #50. The downside is that Rmath-julia is maintained as patches against upstream R, and a larger patch is more painful to rebase.Before any of this, it would be good if @mchitre could confirm the diagnosis, e.g. by running with
LD_PRELOAD=<julia>/lib/julia/libgomp.so.1. Preloaded modules are part of the initial static TLS block, so libgomp would always win the race. This is also inherited by the precompilation worker processes, so it should be a usable workaround for the deployment in the meantime. On glibc >= 2.32,GLIBC_TUNABLES=glibc.rtld.optional_static_tls=4096should have the same effect.If we reorder the imports here anyway, we should at least add a comment and a test that asserts that the SpecialFunctions (or an explicit CompilerSupportLibraries_jll) import comes before the Rmath import. Otherwise we will just lose the invariant again the next time someone touches the module body, exactly as in Remove Rmath sampling methods #188.
Dropping the Rmath dependency here is already in progress independently of this issue (Native implementations for distrs #37, and Replace Rmath hypergeometric with pure Julia implementation #203, Implement beta/gamma invlogcdf and invlogccdf in pure Julia #204, Implement binomial, Poisson, and negative binomial inverse functions in pure Julia #205, Replace Rmath noncentral beta with pure Julia implementation #206 among others), so for StatsFuns users the problem will eventually disappear on its own. It does not help anyone who hits this outside of StatsFuns though, which is why I think 2 is the most immediately useful thing to pursue.
nalimilan commentedon Jul 29, 2026
Thanks for the detailed investigation. Option 2 seems like the most robust solution, but it requires some work which would only be useful for older glibc versions. Option 4 is much cheaper. I guess it depends on whether @mchitre wants to devote some time to it or not.
mchitre commentedon Jul 30, 2026
My current problem is resolved by dropping
StatsFunsas a dependency in our project. The dependency originally came fromDistributions, and that was only used in a relatively small part of the code base that we refactored to drop the dependency.Having said that, if there is anything I can do to help resolve this for the community, I am happy to help.
The
LD_PRELOAD=solution did not work for us because it is not honored withsetcapon thejuliabinary, and we require that.devmotion commentedon Jul 30, 2026
I took a stab at it in JuliaStats/Rmath-julia#57 and JuliaStats/Rmath-julia#58 (based on the former).
devmotion commentedon Aug 3, 2026
Rmath_jll0.5.2+0, built from JuliaStats/Rmath-julia#58, fixes this.The shipped artifacts have a 16-byte TLS segment instead of 856 (
PT_TLSmemsz,aarch64-linux-gnu; 12–16 bytes on the other Linux targets).libgompneeds 136 bytes of initial-exec TLS, so this frees up ~840 bytes of the 1664-byte pre-2.32 surplus.I checked in an
ubuntu:18.04aarch64 container (glibc 2.27, Julia 1.11.3 and 1.11.9, StatsFuns 2.2.1,CompilerSupportLibraries_jll1.1.1+0) with onlyRmath_jlldiffering: with 0.5.1+0 bothusing Rmath, CompilerSupportLibraries_jllandusing StatsFunsfail with the reportedcannot allocate memory in static TLS blockwhile the reversed order works, with 0.5.2+0 everything loads. d/p/q values and sampler streams are byte-identical between the two versions.Rmath0.9 allowsRmath_jll0.4–0.5, so a fresh resolve picks 0.5.2+0 but an existing manifest needs an explicit]up.Thanks @mchitre for the report. The load order here does not matter anymore, so closing.
mchitre commentedon Aug 4, 2026
Thanks all, for helping resolve!