Skip to content

Add free-threaded CPython (3.14t) support - #858

Merged
d-v-b merged 6 commits into
zarr-developers:mainfrom
kumaraditya303:free-threading
Sep 16, 2026
Merged

d-v-b merged 6 commits into
zarr-developers:mainfrom
kumaraditya303:free-threading

Conversation

@kumaraditya303

@kumaraditya303 kumaraditya303 commented Sep 9, 2026 •

Copy link
Copy Markdown
Contributor

Hello, I am a CPython Core Developer working on free-threading support. This PR adds support for free-threading on 3.14t+ (#749).

Changes:

  • Mark all Cython extensions with freethreading_compatible=True so the interpreter no longer re-enables the GIL when numcodecs is imported
  • Require Cython>=3.1, needed for the freethreading_compatible directive
  • blosc: replace the lazily created multiprocessing.Lock (and get_mutex()) with a module-level threading.Lock, reset in forked children via os.register_at_fork
  • blosc: take the lock in _init, _destroy, set_nthreads and the global context path of decompress, so blosc_set_nthreads cannot re-create the global context while another thread is using it
  • blosc: compute the compressor list once at import; blosc_list_compressors() fills a static buffer without locking and was called on every compress()
  • CI: add 3.14t to the test matrix, installing without pcodec, zfpy and google_crc32c which do not ship free-threaded wheels yet
  • Wheels: enable cpython-freethreading in cibuildwheel, build cp314t-* wheels, and assert in the wheel test that the GIL is not re-enabled on free-threaded builds

TODO:

  • Unit tests and/or doctests in docstrings
  • Tests pass locally
  • Docstrings and API docs for any new/modified user-facing classes and functions
  • Changes documented in docs/release.rst
  • Docs build locally
  • GitHub Actions CI passes
  • Test coverage to 100% (Codecov passes)

- Mark all Cython extensions with freethreading_compatible=True so the
  interpreter no longer re-enables the GIL when numcodecs is imported
- Require Cython>=3.1, needed for the freethreading_compatible directive
- blosc: replace the lazily created multiprocessing.Lock (and get_mutex())
  with a module-level threading.Lock, reset in forked children via
  os.register_at_fork
- blosc: take the lock in _init, _destroy, set_nthreads and the global
  context path of decompress, so blosc_set_nthreads cannot re-create the
  global context while another thread is using it
- blosc: compute the compressor list once at import; blosc_list_compressors()
  fills a static buffer without locking and was called on every compress()
- CI: add 3.14t to the test matrix, installing without pcodec, zfpy and
  google_crc32c which do not ship free-threaded wheels yet
- Wheels: enable cpython-freethreading in cibuildwheel, build cp314t-*
  wheels, and assert in the wheel test that the GIL is not re-enabled on
  free-threaded builds
@codecov

codecov Bot commented Sep 9, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (82c3635) to head (2b75ae7).

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #858   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           27        27           
  Lines          905       905           
=========================================
  Hits           905       905           
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@kumaraditya303

Copy link
Copy Markdown
Contributor Author

I dropped -l from shell so that setup-python's PATH order is respected and free-threaded Python is installed correctly, otherwise it incorrectly picks up gil enabled python on macos 3.14t.

@d-v-b

d-v-b commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

I don't know this part of the code very well so I'm acting as a conduit for Claude here. This is the review from claude:

🤖 AI text below 🤖

Built against free-threaded 3.14.7 locally (macOS arm64, Cython 3.3):

  • import numcodecs leaves the GIL disabled (sys._is_gil_enabled() is False).
  • Full test suite passes on 3.14t (721 passed).
  • Stress loop: 8 threads doing compress/decompress on the global context (use_threads = True) while a 9th thread loops set_nthreads(1..4), run with MallocScribble=1. This branch is clean; main segfaults within a few seconds. Cause: blosc_set_nthreads frees and re-allocates g_global_context and global_comp_mutex without holding the mutex, so c-blosc's own locking cannot protect a concurrent blosc_compress. Taking the Python lock in _init/_destroy/set_nthreads and around the two global-context call sites is the right shape. The *_ctx paths read no mutable globals, so leaving them unlocked is correct. Caching blosc_list_compressors() at import is correct given the static buffer and unguarded "done" flag.

Required before merge:

  1. Release note in docs/release.md. Besides free-threading support, two user-visible changes need spelling out: numcodecs.blosc.get_mutex is removed, and on platforms where multiprocessing.Lock() used to raise (no sem_open), the single-threaded ctx path was previously forced and blosc's internal threads will now be used from the main thread.

  2. The 3.14t CI jobs do not verify the GIL stays disabled. Only the cibuildwheel test command does. If a later change re-enabled the GIL, pytest on 3.14t would still be green. A test that runs import numcodecs in a subprocess and asserts not sys._is_gil_enabled() when sysconfig.get_config_var("Py_GIL_DISABLED") is set would close that gap.

  3. Regression test for the locking. The stress loop above fails on main and takes ~1s. Sketch:

    def test_set_nthreads_races_global_context():
        data = np.arange(200_000, dtype='i4')
        stop = threading.Event()
        errors = []
    
        def worker():
            try:
                while not stop.is_set():
                    enc = blosc.compress(data, b'lz4', 5)
                    assert blosc.decompress(enc) == data.tobytes()
            except Exception as e:
                errors.append(e)
    
        def toggler():
            n = 1
            while not stop.is_set():
                n = 1 + (n % 4)
                blosc.set_nthreads(n)
    
        blosc.use_threads = True
        try:
            threads = [threading.Thread(target=worker) for _ in range(8)]
            threads.append(threading.Thread(target=toggler))
            for t in threads:
                t.start()
            time.sleep(1)
            stop.set()
            for t in threads:
                t.join()
        finally:
            blosc.use_threads = None
        assert not errors

@d-v-b

d-v-b commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

"I" can make the required changes if you don't mind

@kumaraditya303

Copy link
Copy Markdown
Contributor Author

"I" can make the required changes if you don't mind

Yes you can make the changes, they all seem fine to me.

d-v-b and others added 2 commits September 16, 2026 20:11
- docs/release.md: document 3.14t support, the Cython>=3.1 build
  requirement, the removal of numcodecs.blosc.get_mutex, and the
  changed fallback on platforms without multiprocessing.Lock
- tests/test_free_threading.py: assert in a fresh interpreter that
  importing numcodecs does not re-enable the GIL on free-threaded builds
- tests/test_blosc.py: regression test racing compress/decompress on
  the global context against set_nthreads; segfaults on main, passes
  with the new locking

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@d-v-b
d-v-b merged commit eff52c8 into zarr-developers:main Sep 16, 2026
43 checks passed
@kumaraditya303
kumaraditya303 deleted the free-threading branch September 17, 2026 04:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants