From abtem zarr

[gvarnavi]

Adding from_abtem reader, subject to abTEM/abTEM#250 being merged. @TomaSusi

@CSSFrancis feel free to adapt for rosettasciio, but I actually think for quantem's purposes we might simply use this, since it supports arbitrary dataset dimensions.

[TomaSusi]

Awesome :) Will review and merge the required abTEM change soon.

How about in-memory abTEM measurements? Might be often handy to have a function to convert them to quantem datasets.

[gvarnavi]

Sure, something like the following does the trick (tested with DiffractionPatterns, Images, and PolarMeasurements with arbitrary metadata):

def to_quantem(abtem_array):
    """
    Convert an in-memory abTEM measurement into a quantem Dataset.
    Assumes both abtem and quantem are installed.
    """

    from quantem.core.datastructures import Dataset

    def _normalize_unit(unit):
        if unit is None:
            return "pixels"

        UNIT_MAP = {
            "Å": "A",
            "Ångström": "A",
            "Angstrom": "A",
            "1/Å": "A^-1",
            "Å^-1": "A^-1",
            "1/A": "A^-1",
        }

        return UNIT_MAP.get(unit, unit)

    sampling = []
    origin = []
    units = []

    for axis in abtem_array.axes_metadata:
        sampling.append(getattr(axis, "sampling", 1.0))
        units.append(_normalize_unit(getattr(axis, "units", None)))
        origin.append(0.0)

    is_lazy = abtem_array.is_lazy
    array = abtem_array.compute().array if is_lazy else abtem_array.array
    
    dataset = Dataset.from_array(
        array=array,
        name=type(abtem_array).__name__,
        origin=tuple(origin),
        sampling=tuple(sampling),
        units=tuple(units),
        signal_units=getattr(abtem_array, "units", "arb. units"),
    )

    # preserve abTEM metadata if present
    if hasattr(abtem_array, "metadata"):
        dataset._metadata = dict(abtem_array.metadata)

    return dataset

However, this should live in abtem.array.ArrayObject imo, not quantem. Will add to abTEM/abTEM#250.

[TomaSusi]

Okay, I don't mind if we add this to abTEM with an optional quantem dependency, just thought it might not be necessary... but perhaps this is the easiest way.

[gvarnavi]

I mean, you don't have hyperspy or xarray as optional dependencies either. The code will issue an error if it can't import quantem and prompt users to do so. I think that's sufficient.

[TomaSusi]

Right, I meant optional in the sense of a import try and a raised exception, it is true we do this for other external packages, too.

[TomaSusi]

The relevant PR (with the merged intermediate changes) is now abTEM/abTEM#252, sorry for any confusion

[TomaSusi]

The abTEM PR is now merged to main.

[arthurmccray]

Overall looks good, though I didn't run any tests. I have one main comment and one question.

The only issue is I see quite a few linter errors that come from me using basedpyright on "standard" while you're using ty. The main thing I don't like about ty is that it defaults unknowns to Any (though I think they're going to add an option to not do that), and as a result it is overly permissive imo. I added a couple of comments, and the easy fix is just to annotate a couple things to Any.

My question is can we automate testing for functions that interface with a package that isn't a dependency? It would be pretty easy for abtem (or another package in the future) to change something that breaks these functions, but we can't exactly write a pytest for it. I don't think this needs to be addressed here in this PR, but it's something to think about going forward.

[gvarnavi]

My question is can we automate testing for functions that interface with a package that isn't a dependency? It would be pretty easy for abtem (or another package in the future) to change something that breaks these functions, but we can't exactly write a pytest for it. I don't think this needs to be addressed here in this PR, but it's something to think about going forward.

Presumably one would use a separate dependency group, say something like:

[dependency-groups]
test-abtem = [
    { include-group = "test" },
    "abtem",
]

and then run using:

uv sync --locked --group test-abtem
uv run pytest tests

[CSSFrancis]

@gvarnavi if you want a look at a fun github action :)

https://github.com/hyperspy/.github/blob/main/.github/workflows/integration_tests.yml

Bascially this runs all of the extension packages every night as well as development versions of different packages. It also assigns people tasks if the package they manage fails.

[arthurmccray]

lgtm!