Skip to content

Fix lazy loading when using pathlib.Path #345

Open
mikekryjak wants to merge 3 commits intomasterfrom
lazy-load-pathlib
Open

Fix lazy loading when using pathlib.Path #345
mikekryjak wants to merge 3 commits intomasterfrom
lazy-load-pathlib

Conversation

@mikekryjak
Copy link
Copy Markdown
Collaborator

The new lazy loader assumed that all paths are strings. When passed a pathlib.Path, it would fall back to the non-lazy loader. This caused me quite a bit of confusion as some of my simulations started taking forever to load.

There is already a tool called _is_path. This PR makes the lazy loader use this to check, which makes it consistent with the same operation elsewhere in the code.

@bendudson was the intention to still lazy load even for a single file? The code design suggests not (you are checking for a collection of files), but in practice single files will still get lazy loaded in the current implementation.

The path check assumed the path was a string, which would make the dataset load in a non-lazy way if the path was pathlib.Path
Copy link
Copy Markdown
Contributor

@dschwoerer dschwoerer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mikekryjak
Copy link
Copy Markdown
Collaborator Author

Looks like we have another CI failure mystery....

Comment thread xbout/lazyload.py
Comment on lines +354 to +355
if name == "dz":
data_vars[name] = var
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logs show the tests failing due to missing dz -- this is the only change that mentions dz, but I don't see how it could fail (this line doesn't seem to come up in the stacktrace either)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is what I added to fix it :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants