Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatically fall back to fsspec if s3/gcs/adfs path is provided in open_mfdataset #9723

Open
phofl opened this issue Nov 6, 2024 · 4 comments

Comments

@phofl
Copy link
Contributor

phofl commented Nov 6, 2024

Is your feature request related to a problem?

Currently, passing a list of s3 (or any other cloud storage) to open_mfdataset will raise, which is not a great ux. Users have to open the file handles for every file themselves and pass those in

Describe the solution you'd like

I'd like a solution similar to what open_zarr is doing in (_normalize_store_arg_v2), probably factor this out in a helper function and call in both places. Happy to put up a PR for this

cc @dcherian FYI

Describe alternatives you've considered

No response

Additional context

No response

@dcherian
Copy link
Contributor

dcherian commented Nov 6, 2024

We discussed at this meeting today. There was broad support for this idea, but also a request to limit the implementation to relative simple, low-level fsspec API since we might want to swap out fsspec for an alternative in the future.

Can you send in a PR please?

Also turns out zarr does the resolving on its own now, so we might be able to delete xarray's fsspec code path

@phofl
Copy link
Contributor Author

phofl commented Nov 6, 2024

Thanks!

Isolating things to a single helper function would be suitable for this? That would keep the scope limited to a single function that you could swap out whenever

@dcherian
Copy link
Contributor

dcherian commented Nov 6, 2024

Yes that would be great. Thanks!

@jrbourbeau
Copy link
Contributor

I opened up #9797 for this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants