Deadlock when running `dl_iterate_phdr` in parallel with GIL Python

[itamarst]

This should be a release blocker.

I'm testing https://github.com/ogrisel/example_pipelines/ with joblib updated to use threadpoolctl, and unreleased latest threadpoolctl, and getting deadlocks. I'm seeing dl_iterate_phdr in two different threads, both waiting on an internal futex, is suggestive.

A bit more details:

The thread holding the GIL

(/home/itamarst/devel/example_pipelines/.pixi/envs/pypi-gil/lib/python3.13/site-packages/numpy/_core/_multiarray_umath.cpython-313-x86_64-linux-gnu.so)
    (C) File "./debug/backtrace.c", line 78, in __backtrace (/usr/lib/x86_64-linux-gnu/libc.so.6)
    (C) File "../../../libgcc/unwind.inc", line 296, in _Unwind_Backtrace (/home/itamarst/devel/example_pipelines/.pixi/envs/pypi-gil/lib/libgcc_s.so.1)
    (C) File "../../../libgcc/unwind-dw2.c", line 1342, in uw_init_context_1 (/home/itamarst/devel/example_pipelines/.pixi/envs/pypi-gil/lib/libgcc_s.so.1)
    (C) File "../../../libgcc/unwind-dw2.c", line 1008, in uw_frame_state_for (/home/itamarst/devel/example_pipelines/.pixi/envs/pypi-gil/lib/libgcc_s.so.1)
    (C) File "../../../libgcc/unwind-dw2-fde-dip.c", line 569, in _Unwind_Find_FDE (/home/itamarst/devel/example_pipelines/.pixi/envs/pypi-gil/lib/libgcc_s.so.1)
    (C) File "./elf/dl-iteratephdr.c", line 39, in dl_iterate_phdr (/usr/lib/x86_64-linux-gnu/libc.so.6)
    (C) File "./nptl/pthread_mutex_lock.c", line 128, in __pthread_mutex_lock@GLIBC_2.2.5 (/usr/lib/x86_64-linux-gnu/libc.so.6)
    (C) File "./nptl/pthread_mutex_lock.c", line 48, in lll_mutex_lock_optimized (inlined) (/usr/lib/x86_64-linux-gnu/libc.so.6)
    (C) File "./nptl/lowlevellock.c", line 49, in __GI___lll_lock_wait (/usr/lib/x86_64-linux-gnu/libc.so.6)
    (C) File "../sysdeps/nptl/futex-internal.h", line 146, in futex_wait (inlined) (/usr/lib/x86_64-linux-gnu/libc.so.6)

The thread trying to do dl_iterate_phdr

Does not hold GIL.

        libc.dl_iterate_phdr(c_match_library_callback, data)
    (C) File "/usr/local/src/conda/python-3.13.14/Modules/_ctypes/_ctypes.c", line 4399, in PyCFuncPtr_call (/home/itamarst/devel/example_pipelines/.pixi/envs/pypi-gil/lib/python3.13/lib-dynload/_ctypes.cpython-313-x86_64-linux-gnu.so)
    (C) File "/usr/local/src/conda/python-3.13.14/Modules/_ctypes/callproc.c", line 1301, in _ctypes_callproc (/home/itamarst/devel/example_pipelines/.pixi/envs/pypi-gil/lib/python3.13/lib-dynload/_ctypes.cpython-313-x86_64-linux-gnu.so)
    (C) File "/usr/local/src/conda/python-3.13.14/Modules/_ctypes/callproc.c", line 950, in _call_function_pointer (inlined) (/home/itamarst/devel/example_pipelines/.pixi/envs/pypi-gil/lib/python3.13/lib-dynload/_ctypes.cpython-313-x86_64-linux-gnu.so)
    (C) File "???", line 0, in ffi_call (/home/itamarst/devel/example_pipelines/.pixi/envs/pypi-gil/lib/libffi.so.8.2.0)
    (C) File "???", line 0, in ffi_call_int (/home/itamarst/devel/example_pipelines/.pixi/envs/pypi-gil/lib/libffi.so.8.2.0)
    (C) File "???", line 0, in ffi_call_unix64 (/home/itamarst/devel/example_pipelines/.pixi/envs/pypi-gil/lib/libffi.so.8.2.0)
    (C) File "./elf/dl-iteratephdr.c", line 74, in dl_iterate_phdr (/usr/lib/x86_64-linux-gnu/libc.so.6)
    (C) File "???", line 0, in ffi_closure_unix64 (/home/itamarst/devel/example_pipelines/.pixi/envs/pypi-gil/lib/libffi.so.8.2.0)
    (C) File "???", line 0, in ffi_closure_unix64_inner (/home/itamarst/devel/example_pipelines/.pixi/envs/pypi-gil/lib/libffi.so.8.2.0)
    (C) File "/usr/local/src/conda/python-3.13.14/Modules/_ctypes/callbacks.c", line 299, in closure_fcn (/home/itamarst/devel/example_pipelines/.pixi/envs/pypi-gil/lib/python3.13/lib-dynload/_ctypes.cpython-313-x86_64-linux-gnu.so)
    (C) File "/usr/local/src/conda/python-3.13.14/Python/pystate.c", line 2796, in PyGILState_Ensure (/home/itamarst/devel/example_pipelines/.pixi/envs/pypi-gil/bin/python3.13)
    (C) File "/usr/local/src/conda/python-3.13.14/Python/ceval_gil.c", line 337, in take_gil (/home/itamarst/devel/example_pipelines/.pixi/envs/pypi-gil/bin/python3.13)
    (C) File "/usr/local/src/conda/python-3.13.14/Python/condvar.h", line 74, in PyCOND_TIMEDWAIT (inlined) (/home/itamarst/devel/example_pipelines/.pixi/envs/pypi-gil/bin/python3.13)
    (C) File "./nptl/pthread_cond_wait.c", line 652, in pthread_cond_timedwait@@GLIBC_2.3.2 (/usr/lib/x86_64-linux-gnu/libc.so.6)
    (C) File "./nptl/pthread_cond_wait.c", line 503, in __pthread_cond_wait_common (inlined) (/usr/lib/x86_64-linux-gnu/libc.so.6)
    (C) File "./nptl/futex-internal.c", line 139, in __GI___futex_abstimed_wait_cancelable64 (/usr/lib/x86_64-linux-gnu/libc.so.6)
    (C) File "./nptl/futex-internal.c", line 87, in __futex_abstimed_wait_common (inlined) (/usr/lib/x86_64-linux-gnu/libc.so.6)
    (C) File "./nptl/futex-internal.c", line 57, in __futex_abstimed_wait_common64 (inlined) (/usr/lib/x86_64-linux-gnu/libc.so.6)

A theory

The thread doing dl_iterate_phdr is trying to acquire GIL so it can do Python callback. Probably it is holding on to some internal dl system lock.

The thread holding GIL somehow triggers dl_iterate_phdr via gcc's unwind support (why? how?). Which means it tries to acquires the dl lock.

Thus, a deadlock.

On free-threaded Python there'd be no GIL so probably no deadlock?

A potential solution

This seems to solve the problem: load libc with ctypes.PyDLL instead of ctypes.CDLL, so it doesn't release the GIL to do dl_iterate_phdr.

Next step is figuring out if I can make a minimal reproducer to turn into a threadpoolctl.py test.

[ogrisel]

@itamarst I just discovered that dl_iterate_phdr can trigger another kind of problem (#225) on some Linux distros with an older libffi. Maybe you can check if the proposed workaround for that bug (#242) can help you solve the GIL deadlock.

EDIT: ctypes.CDLL is still used in #242 though. So maybe that won't help.

[itamarst]

Thank you, will take a look. Further experiments suggest maybe my hypothesis was incorrect, or insufficient, since PyDLL doesn't always fix it (maybe makes it less likely? maybe there's two deadlocks? Haven't figured out yet).

Separately I've discovered that Python 3.14 has ctypes.utils.dllist which implements all this logic for you, so seems useful to use assuming it doesn't have the same problems (it keeps the GIL held, at least).