Skip to content

FIX FlexiBLAS CI default backend and scope script hang - #296

Merged
ogrisel merged 2 commits into
joblib:masterfrom
ogrisel:debug-ci
Nov 5, 2026
Merged

ogrisel merged 2 commits into
joblib:masterfrom
ogrisel:debug-ci

Conversation

@ogrisel

@ogrisel ogrisel commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Fixes the FlexiBLAS CI jobs, which started failing on master after an upstream FlexiBLAS change. Found while investigating the failures in #294 (#294).

Root cause

  • Upstream FlexiBLAS (commits 024e3c19c and 28cf5ba3e, Nov 3 2026) made the default backend a CMake option named FLEXIBLAS_DEFAULT_BLAS. continuous_integration/install_flexiblas.sh still passed -DFLEXIBLAS_DEFAULT=OPENBLAS_CONDA, which is now silently ignored. FlexiBLAS falls back to NETLIB, so test_flexiblas and test_flexiblas_switch fail on Linux.
  • In tests/empirical_scope_observation.py, run() starts a non-daemon thread that counts threads and only stops it when the BLAS/OpenMP probe returns normally. If the probe raises (on macOS, numpy's import-time sanity check failed under the NETLIB default), the thread keeps the interpreter alive and the job hangs until the 30 minute timeout.

Changes

  • Pass -DFLEXIBLAS_DEFAULT_BLAS="OPENBLAS_CONDA".
  • Fail the FlexiBLAS install step if the generated config does not contain default = OPENBLAS_CONDA, so a future silent fallback breaks the build immediately.
  • Stop the thread counter in a finally block so an exception in the probe cannot hang the process.

Evidence

  • Before (fa70d83, and FIX recursive wrap, plus controlling thread cleanups and doc improvements #294 at b57a863): Linux FlexiBLAS reports Default BLAS: NETLIB and fails the two tests. macOS hangs. The nightly master schedule failed the same jobs on Nov 5.
  • After (3224de8): all 26 checks pass. Both FlexiBLAS jobs report Default BLAS: OPENBLAS_CONDA and current_backend: OPENBLAS_CONDA.
  • Locally, the old scope script hangs when the probe raises and the patched version exits with the error.

Open questions

  • Why numpy's sanity check fails on macOS with the NETLIB backend is not investigated here. It does not affect the setup once OPENBLAS_CONDA is the default.
  • FlexiBLAS is still built from its master branch. Pinning a release tag would avoid this kind of breakage, at the cost of catching upstream regressions later.

ogrisel and others added 2 commits November 5, 2026 08:14
…pe observation script

Co-authored-by: Cursor <cursoragent@cursor.com>
@ogrisel
ogrisel marked this pull request as ready for review November 5, 2026 08:01
@ogrisel ogrisel changed the title Investigate FlexiBLAS CI failures FIX FlexiBLAS CI default backend and scope script hang Nov 5, 2026
@ogrisel
ogrisel marked this pull request as draft November 5, 2026 08:01
@ogrisel
ogrisel marked this pull request as ready for review November 5, 2026 09:08

@ogrisel ogrisel left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I will merge.

@ogrisel
ogrisel merged commit abcd95b into joblib:master Nov 5, 2026
26 checks passed
@ogrisel
ogrisel deleted the debug-ci branch November 5, 2026 09:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant