Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compatibility for zarr-python 3.x #9552

Merged
merged 93 commits into from
Oct 23, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
93 commits
Select commit Hold shift + click to select a range
c54a052
Remove zarr pin
TomAugspurger Sep 27, 2024
483eb7f
Define zarr_v3 helper
TomAugspurger Sep 23, 2024
40a746c
zarr-v3: filters / compressors -> codecs
TomAugspurger Sep 23, 2024
6c8d2bb
zarr-v3: update tests to avoid values equal to fillValue
TomAugspurger Sep 23, 2024
531b521
Various test fixes
TomAugspurger Sep 27, 2024
849df40
zarr_version fixes
TomAugspurger Sep 30, 2024
88bd64b
fixup! zarr-v3: filters / compressors -> codecs
TomAugspurger Sep 30, 2024
ef1549a
fixup! fixup! zarr-v3: filters / compressors -> codecs
TomAugspurger Sep 30, 2024
20c22bd
fixup
TomAugspurger Sep 30, 2024
6087e5e
path / key normalization in set_variables
TomAugspurger Sep 30, 2024
15fe55e
fixes
TomAugspurger Oct 1, 2024
8e06bc7
workaround nested consolidated metadata
TomAugspurger Oct 1, 2024
f22100e
Merge remote-tracking branch 'upstream/main' into fix/zarr-v3
TomAugspurger Oct 1, 2024
f8c427f
test: avoid fill_value
TomAugspurger Oct 1, 2024
594d36d
test: Adjust call counts
TomAugspurger Oct 2, 2024
046d37e
zarr-python 3.x Array.resize doesn't mutate
TomAugspurger Oct 2, 2024
6b0ca62
test compatibility
TomAugspurger Oct 2, 2024
d315583
skip ZipStore with_mode
TomAugspurger Oct 2, 2024
389cc82
test: more fill_value avoidance
TomAugspurger Oct 2, 2024
1fe409a
test: more fill_value avoidance
TomAugspurger Oct 2, 2024
7c29ea6
v3 compat for instrumented test
TomAugspurger Oct 2, 2024
efb66dd
Merge remote-tracking branch 'upstream/main' into fix/zarr-v3
TomAugspurger Oct 7, 2024
90c0ae6
Merge remote-tracking branch 'upstream/main' into fix/zarr-v3
TomAugspurger Oct 8, 2024
9b3c288
Handle zarr_version / zarr_format deprecation
TomAugspurger Oct 8, 2024
3717391
wip
rabernat Oct 9, 2024
8d16bb2
most Zarr tests passing
rabernat Oct 9, 2024
bd978b0
unskip tests
rabernat Oct 9, 2024
34c4c24
add custom Zarr _FillValue encoding / decoding
rabernat Oct 9, 2024
118e50e
Merge branch 'fix/zarr-v3' into ryan/fix/zarr-3
TomAugspurger Oct 9, 2024
e6e2066
Merge pull request #1 from rabernat/ryan/fix/zarr-3
TomAugspurger Oct 9, 2024
1d1d9cb
relax dtype comparison in test_roundtrip_empty_vlen_string_array
rabernat Oct 9, 2024
9089508
Merge remote-tracking branch 'tom/fix/zarr-v3' into ryan/fix/zarr-3
rabernat Oct 9, 2024
ea00308
fix test_explicitly_omit_fill_value_via_encoding_kwarg
rabernat Oct 9, 2024
a330e4b
fix test_append_string_length_mismatch_raises
rabernat Oct 9, 2024
bde42ee
fix test_check_encoding_is_consistent_after_append for v3
rabernat Oct 10, 2024
b15705d
skip roundtrip_endian for zarr v3
rabernat Oct 10, 2024
38f43b9
unskip datetimes and fix test_compressor_encoding
rabernat Oct 10, 2024
1cfc458
unskip tests
rabernat Oct 10, 2024
af1a0b8
add back dtype skip
rabernat Oct 10, 2024
d9d6fee
Merge branches 'ryan/fix/zarr-v3-2', 'ryan/fix/zarr-v3-3', 'ryan/fix/…
rabernat Oct 10, 2024
4c54371
Merge pull request #2 from rabernat/ryan/fix/zarr-v3-2
TomAugspurger Oct 10, 2024
1ce8878
Merge pull request #9 from rabernat/ryan/fix/zarr-v3-combined
TomAugspurger Oct 10, 2024
0c2e260
point upstream to v3 branch
TomAugspurger Oct 10, 2024
fc2738a
Merge branch 'main' into fix/zarr-v3
jhamman Oct 11, 2024
0e47c3f
Create temporary directory before using it
TomAugspurger Oct 11, 2024
5b39f42
Avoid zarr.storage.zip on v2
TomAugspurger Oct 11, 2024
7d9fc05
fixed close_store_on_close bug
TomAugspurger Oct 11, 2024
0fa94ee
Remove workaround, fixed upstream
TomAugspurger Oct 11, 2024
c2fd6f1
Restore original `w` mode.
TomAugspurger Oct 11, 2024
ac2ef29
workaround for store closing with mode=w
TomAugspurger Oct 11, 2024
c6be467
typing fixes
TomAugspurger Oct 11, 2024
5b5b77f
compat
TomAugspurger Oct 11, 2024
4f07eb7
Remove unnecessary pop
TomAugspurger Oct 11, 2024
5151bc2
fixed skip
TomAugspurger Oct 11, 2024
00c62d7
fixup types
TomAugspurger Oct 11, 2024
e0390a5
fixup types
TomAugspurger Oct 11, 2024
2e7ec07
[test-upstream]
TomAugspurger Oct 12, 2024
26081d4
Update install-upstream-wheels.sh
jhamman Oct 12, 2024
0350056
set use_consolidated to false when user provides consolidated=False
jhamman Oct 13, 2024
a38bff6
fix: import consolidated_metadata from package root
jhamman Oct 14, 2024
08f0594
fix: relax instrumented store checks for v3
jhamman Oct 14, 2024
0e81edf
Merge pull request #10 from jhamman/fix/zarr-v3-consolidated-false
TomAugspurger Oct 14, 2024
55d852d
Merge pull request #11 from jhamman/fix/cm-import
TomAugspurger Oct 14, 2024
3491137
Merge pull request #12 from jhamman/fix/instrumented-store-check
TomAugspurger Oct 14, 2024
5bf5f2a
Merge remote-tracking branch 'upstream/main' into fix/zarr-v3
TomAugspurger Oct 14, 2024
f2f9fff
Adjust 2.18.3 thresholds
TomAugspurger Oct 14, 2024
a84fa79
skip datatree zarr tests w/ zarr 3 for now
jhamman Oct 14, 2024
c280f24
Merge pull request #13 from jhamman/fix/skip-datatree-tests
TomAugspurger Oct 14, 2024
9fec1d6
fixed kvstore usage
TomAugspurger Oct 14, 2024
04c017e
typing fixes
TomAugspurger Oct 14, 2024
625591e
move zarr.codecs import
TomAugspurger Oct 14, 2024
3795b07
fixup ignores
TomAugspurger Oct 14, 2024
ea2cb57
storage options fix, skip
TomAugspurger Oct 14, 2024
4f617d2
fixed types
TomAugspurger Oct 14, 2024
d1e3c73
Update ci/install-upstream-wheels.sh
jhamman Oct 14, 2024
45d5a78
type fixes
jhamman Oct 14, 2024
f208c39
whats-new
jhamman Oct 14, 2024
968217c
Update xarray/tests/test_backends_datatree.py
jhamman Oct 14, 2024
45a37f6
fix type import
jhamman Oct 14, 2024
c15e856
Merge branch 'fix/zarr-v3' of github.com:TomAugspurger/xarray into fi…
jhamman Oct 14, 2024
c10bfc0
set mapper, chunk_mapper
TomAugspurger Oct 15, 2024
82e6a6d
Pass through zarr_format
TomAugspurger Oct 15, 2024
d752693
Fixup
TomAugspurger Oct 15, 2024
0fd4103
Merge remote-tracking branch 'upstream/main' into fix/zarr-v3
TomAugspurger Oct 15, 2024
c2a47a1
more cleanup
TomAugspurger Oct 15, 2024
26b2661
revert test changes
TomAugspurger Oct 15, 2024
1d73d36
Update xarray/backends/zarr.py
dcherian Oct 21, 2024
be79e88
cleanup
dcherian Oct 21, 2024
ff0f2c0
update docstring
dcherian Oct 21, 2024
5f37042
Merge branch 'main' into fix/zarr-v3
dcherian Oct 21, 2024
268e3eb
fix rtd
dcherian Oct 22, 2024
1abb2ba
Merge branch 'main' into fix/zarr-v3
dcherian Oct 22, 2024
7682bf4
tweak
dcherian Oct 22, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions ci/install-upstream-wheels.sh
Original file line number Diff line number Diff line change
Expand Up @@ -45,15 +45,15 @@ python -m pip install \
--pre \
--upgrade \
pyarrow
# manually install `pint` to pull in new dependencies
python -m pip install --upgrade pint
# manually install `pint`, `donfig`, and `crc32c` to pull in new dependencies
python -m pip install --upgrade pint donfig crc32c
python -m pip install \
--no-deps \
--upgrade \
git+https://github.com/dask/dask \
git+https://github.com/dask/dask-expr \
git+https://github.com/dask/distributed \
git+https://github.com/zarr-developers/zarr.git@main \
git+https://github.com/zarr-developers/zarr \
git+https://github.com/Unidata/cftime \
git+https://github.com/pypa/packaging \
git+https://github.com/hgrecco/pint \
Expand Down
3 changes: 2 additions & 1 deletion doc/user-guide/io.rst
Original file line number Diff line number Diff line change
Expand Up @@ -823,8 +823,9 @@ For example:
.. ipython:: python

import zarr
from numcodecs.blosc import Blosc
TomAugspurger marked this conversation as resolved.
Show resolved Hide resolved

compressor = zarr.Blosc(cname="zstd", clevel=3, shuffle=2)
compressor = Blosc(cname="zstd", clevel=3, shuffle=2)
ds.to_zarr("foo.zarr", encoding={"foo": {"compressor": compressor}})

.. note::
Expand Down
4 changes: 4 additions & 0 deletions doc/whats-new.rst
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,10 @@ New Features
By `Holly Mandel <https://github.com/hollymandel>`_.
- Implement handling of complex numbers (netcdf4/h5netcdf) and enums (h5netcdf) (:issue:`9246`, :issue:`3297`, :pull:`9509`).
By `Kai Mühlbauer <https://github.com/kmuehlbauer>`_.
- Support for Zarr-Python 3 (:issue:`95515`, :pull:`9552`).
By `Tom Augspurger <https://github.com/TomAugspurger>`_,
`Ryan Abernathey <https://github.com/rabernat>`_ and
`Joe Hamman <https://github.com/jhamman>`_.

Breaking changes
~~~~~~~~~~~~~~~~
Expand Down
3 changes: 2 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ dev = [
"sphinx_autosummary_accessors",
"xarray[complete]",
]
io = ["netCDF4", "h5netcdf", "scipy", 'pydap; python_version<"3.10"', "zarr<3", "fsspec", "cftime", "pooch"]
io = ["netCDF4", "h5netcdf", "scipy", 'pydap; python_version<"3.10"', "zarr", "fsspec", "cftime", "pooch"]
etc = ["sparse"]
parallel = ["dask[complete]"]
viz = ["cartopy", "matplotlib", "nc-time-axis", "seaborn"]
Expand Down Expand Up @@ -124,6 +124,7 @@ module = [
"nc_time_axis.*",
"netCDF4.*",
"netcdftime.*",
"numcodecs.*",
"opt_einsum.*",
"pint.*",
"pooch.*",
Expand Down
30 changes: 17 additions & 13 deletions xarray/backends/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
_normalize_path,
)
from xarray.backends.locks import _get_scheduler
from xarray.backends.zarr import _zarr_v3
from xarray.core import indexing
from xarray.core.combine import (
_infer_concat_order_from_positions,
Expand Down Expand Up @@ -1685,6 +1686,7 @@ def to_zarr(
safe_chunks: bool = True,
storage_options: dict[str, str] | None = None,
zarr_version: int | None = None,
zarr_format: int | None = None,
write_empty_chunks: bool | None = None,
chunkmanager_store_kwargs: dict[str, Any] | None = None,
) -> backends.ZarrStore | Delayed:
Expand All @@ -1703,21 +1705,28 @@ def to_zarr(
store = _normalize_path(store)
chunk_store = _normalize_path(chunk_store)

kwargs = {}
if storage_options is None:
mapper = store
chunk_mapper = chunk_store
else:
from fsspec import get_mapper

if not isinstance(store, str):
raise ValueError(
f"store must be a string to use storage_options. Got {type(store)}"
)
mapper = get_mapper(store, **storage_options)
if chunk_store is not None:
chunk_mapper = get_mapper(chunk_store, **storage_options)
else:

if _zarr_v3():
kwargs["storage_options"] = storage_options
TomAugspurger marked this conversation as resolved.
Show resolved Hide resolved
mapper = store
chunk_mapper = chunk_store
else:
from fsspec import get_mapper

mapper = get_mapper(store, **storage_options)
if chunk_store is not None:
chunk_mapper = get_mapper(chunk_store, **storage_options)
else:
chunk_mapper = chunk_store

if encoding is None:
encoding = {}
Expand Down Expand Up @@ -1747,13 +1756,6 @@ def to_zarr(
# validate Dataset keys, DataArray names
_validate_dataset_names(dataset)

if zarr_version is None:
# default to 2 if store doesn't specify its version (e.g. a path)
zarr_version = int(getattr(store, "_store_version", 2))

if consolidated is None and zarr_version > 2:
consolidated = False

if mode == "r+":
already_consolidated = consolidated
consolidate_on_close = False
Expand All @@ -1773,7 +1775,9 @@ def to_zarr(
safe_chunks=safe_chunks,
stacklevel=4, # for Dataset.to_zarr()
zarr_version=zarr_version,
zarr_format=zarr_format,
write_empty=write_empty_chunks,
**kwargs,
)

if region is not None:
Expand Down
Loading
Loading