Scope does not work with Python 3.12 generics

[doronbehar]

Test failures we experience on NixOS:

============================= test session starts ==============================
platform linux -- Python 3.14.6, pytest-9.1.1, pluggy-1.6.0 -- /nix/store/rgnappqqc5vbq60gza5fflyk84sylwl6-python3-3.14.6/bin/python3.14
cachedir: .pytest_cache
Using --randomly-seed=17394782
rootdir: /build/source
configfile: pyproject.toml
testpaths: tests
plugins: randomly-4.0.1
collected 231 items / 1 error

==================================== ERRORS ====================================
_______________ ERROR collecting tests/complex_workflow_test.py ________________
tests/complex_workflow_test.py:32: in <module>
    class Masked(sl.Scope[Run, npt.NDArray[np.float64]], npt.NDArray[np.float64]): ...
E   TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
=========================== short test summary info ============================
ERROR tests/complex_workflow_test.py - TypeError: metaclass conflict: the metaclass of a derived class must be a (...
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.85s ===============================

Even when we added the --ignore-glob=tests/complex_workflow_test.py Pytest flag, we got the error:

=================================== FAILURES ===================================
______ test_subclasses_of_generic_array_provider_defined_with_Scope_work _______

    def test_subclasses_of_generic_array_provider_defined_with_Scope_work() -> None:
        # int is unused, but a single constraint is not allowed by Python
        Param = TypeVar('Param', str, int)
    
>       class ArrayT(sl.Scope[Param, npt.NDArray[np.int64]], npt.NDArray[np.int64]): ...
E       TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

tests/pipeline_test.py:256: TypeError
=========================== short test summary info ============================
SKIPPED [1] tests/pipeline_test.py:1269: Test requires dask.distributed
SKIPPED [1] tests/pipeline_test.py:1286: Test requires dask.distributed
SKIPPED [1] tests/pipeline_test.py:1259: Test requires dask.distributed
FAILED tests/pipeline_test.py::test_subclasses_of_generic_array_provider_defined_with_Scope_work - TypeError: metaclass conflict: the metaclass of a derived class must be a (...
=================== 1 failed, 227 passed, 3 skipped in 3.03s ===================

[jl-wynen]

This is a clash between typing.Generic and Python 3.12's type aliases. E.g.,:

from typing import Generic

type A = list[str]
class M(Generic[Run], A): ...

I didn't dig too deep but NumPy seems to have changed to using the new type aliases. (I tried numpy >= 2.4)

[jl-wynen]

This will affect any code that uses Python 3.12 generics. The most straightforward fix is to avoid npt.NDArray:

class Masked(sl.Scope[Run, np.ndarray[np.float64]], np.ndarray[np.float64]): ...

However, we can avoid Scope altogether using (using [np.float64] here is optional)

class Masked[Run](np.ndarray[np.float64]): ...

But note that this Run type parameter is unconstrained and unrelated to the Run TypeVar used elsewhere. Further, it cannot be constrained using the constraints argument of Pipeline.__init__. So we may not want to use this approach.

I'll fix the tests but we may want to discuss how to handle generics going forward and if we need a guide because I can see this problem appearing in workflows.

[jl-wynen]

@doronbehar There is a new release with fixed tests.