Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Conformant ZarrV3 codecs and fill values #193

Merged
merged 79 commits into from
Jul 22, 2024
Merged

Conformant ZarrV3 codecs and fill values #193

merged 79 commits into from
Jul 22, 2024

Commits on Apr 19, 2024

  1. Generate chunk manifest backed variable from HDF5 dataset.

    @sharkinsspatial
    sharkinsspatial committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    6b7abe2 View commit details
    Browse the repository at this point in the history

  2. Transfer dataset attrs to variable.

    @sharkinsspatial
    sharkinsspatial committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    bca0aab View commit details
    Browse the repository at this point in the history

  3. Get virtual variables dict from HDF5 file.

    @sharkinsspatial
    sharkinsspatial committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    384ff6b View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2024

  1. Update virtual_vars_from_hdf to use fsspec and drop_variables arg.

    @sharkinsspatial
    sharkinsspatial committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    4c5f9bd View commit details
    Browse the repository at this point in the history

  2. mypy fix to use ChunkKey and empty dimensions list.

    @sharkinsspatial
    sharkinsspatial committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    1dd3370 View commit details
    Browse the repository at this point in the history

  3. Extract attributes from hdf5 root group.

    @sharkinsspatial
    sharkinsspatial committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    d92c75c View commit details
    Browse the repository at this point in the history

  4. Use hdf reader for netcdf4 files.

    @sharkinsspatial
    sharkinsspatial committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    0ed8362 View commit details
    Browse the repository at this point in the history

  5. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    f4485fa View commit details
    Browse the repository at this point in the history

Commits on May 8, 2024

  1. Merge branch 'main' into hdf5_reader

    @sharkinsspatial
    sharkinsspatial committed May 8, 2024
    Configuration menu
    Copy the full SHA
    3cc1254 View commit details
    Browse the repository at this point in the history

Commits on May 9, 2024

  1. Fix ruff complaints.

    @sharkinsspatial
    sharkinsspatial committed May 9, 2024
    Configuration menu
    Copy the full SHA
    0123df7 View commit details
    Browse the repository at this point in the history

Commits on May 10, 2024

  1. First steps for handling HDF5 filters.

    @sharkinsspatial
    sharkinsspatial committed May 10, 2024
    Configuration menu
    Copy the full SHA
    332bcaa View commit details
    Browse the repository at this point in the history

Commits on May 13, 2024

  1. Initial step for hdf5plugin supported codecs.

    @sharkinsspatial
    sharkinsspatial committed May 13, 2024
    Configuration menu
    Copy the full SHA
    c51e615 View commit details
    Browse the repository at this point in the history

Commits on May 16, 2024

  1. Small commit to check compression support in CI environment.

    @sharkinsspatial
    sharkinsspatial committed May 16, 2024
    Configuration menu
    Copy the full SHA
    0083f77 View commit details
    Browse the repository at this point in the history

Commits on May 18, 2024

  1. Merge branch 'main' into hdf5_reader

    @sharkinsspatial
    sharkinsspatial committed May 18, 2024
    Configuration menu
    Copy the full SHA
    3c00071 View commit details
    Browse the repository at this point in the history

Commits on May 19, 2024

  1. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed May 19, 2024
    Configuration menu
    Copy the full SHA
    207c4b5 View commit details
    Browse the repository at this point in the history

  2. Fix mypy complaints for hdf_filters.

    @sharkinsspatial
    sharkinsspatial committed May 19, 2024
    Configuration menu
    Copy the full SHA
    c573800 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'hdf5_reader' of https://github.com/TomNicholas/Virtuali… 

    …Zarr into hdf5_reader
    @sharkinsspatial
    sharkinsspatial committed May 19, 2024
    Configuration menu
    Copy the full SHA
    ef0d7a8 View commit details
    Browse the repository at this point in the history

  4. Local pre-commit fix for hdf_filters.

    @sharkinsspatial
    sharkinsspatial committed May 19, 2024
    Configuration menu
    Copy the full SHA
    588e06b View commit details
    Browse the repository at this point in the history

Commits on May 21, 2024

  1. Use fsspec reader_options introduced in zarr-developers#37.

    @sharkinsspatial
    sharkinsspatial committed May 21, 2024
    Configuration menu
    Copy the full SHA
    725333e View commit details
    Browse the repository at this point in the history

  2. Fix incorrect zarr_v3 if block position from merge commit ef0d7a8.

    @sharkinsspatial
    sharkinsspatial committed May 21, 2024
    Configuration menu
    Copy the full SHA
    72df108 View commit details
    Browse the repository at this point in the history

  3. Fix early return from hdf _extract_attrs.

    @sharkinsspatial
    sharkinsspatial committed May 21, 2024
    Configuration menu
    Copy the full SHA
    d1e85cb View commit details
    Browse the repository at this point in the history

  4. Test that _extract_attrs correctly handles multiple attributes.

    @sharkinsspatial
    sharkinsspatial committed May 21, 2024
    Configuration menu
    Copy the full SHA
    1e2b343 View commit details
    Browse the repository at this point in the history

Commits on May 22, 2024

  1. Initial attempt at scale and offset via numcodecs.

    @sharkinsspatial
    sharkinsspatial committed May 22, 2024
    Configuration menu
    Copy the full SHA
    7f1c189 View commit details
    Browse the repository at this point in the history

Commits on May 23, 2024

  1. Tests for cfcodec_from_dataset.

    @sharkinsspatial
    sharkinsspatial committed May 23, 2024
    Configuration menu
    Copy the full SHA
    908e332 View commit details
    Browse the repository at this point in the history

Commits on May 24, 2024

  1. Temporarily relax integration tests to assert_allclose.

    @sharkinsspatial
    sharkinsspatial committed May 24, 2024
    Configuration menu
    Copy the full SHA
    0df332d View commit details
    Browse the repository at this point in the history

  2. Add blosc_lz4 fixture parameterization to confirm libnetcdf environment.

    @sharkinsspatial
    sharkinsspatial committed May 24, 2024
    Configuration menu
    Copy the full SHA
    ca6b236 View commit details
    Browse the repository at this point in the history

  3. Check for compatability with netcdf4 engine.

    @sharkinsspatial
    sharkinsspatial committed May 24, 2024
    Configuration menu
    Copy the full SHA
    b7426c5 View commit details
    Browse the repository at this point in the history

Commits on May 27, 2024

  1. Use separate fixtures for h5netcdf and netcdf4 compression styles.

    @sharkinsspatial
    sharkinsspatial committed May 27, 2024
    Configuration menu
    Copy the full SHA
    dac21dd View commit details
    Browse the repository at this point in the history

  2. Print libhdf5 and libnetcdf4 versions to confirm compiled environment.

    @sharkinsspatial
    sharkinsspatial committed May 27, 2024
    Configuration menu
    Copy the full SHA
    e968772 View commit details
    Browse the repository at this point in the history

  3. Skip netcdf4 style compression tests when libhdf5 < 1.14.

    @sharkinsspatial
    sharkinsspatial committed May 27, 2024
    Configuration menu
    Copy the full SHA
    9a98e57 View commit details
    Browse the repository at this point in the history

Commits on Jun 11, 2024

  1. Include imagecodecs.numcodecs to support HDF5 lzf filters.

    @sharkinsspatial
    sharkinsspatial committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    7590b87 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into hdf5_reader

    @sharkinsspatial
    sharkinsspatial committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    e9fbc8a View commit details
    Browse the repository at this point in the history

  3. Remove test that verifies call to read_kerchunk_references_from_file.

    @sharkinsspatial
    sharkinsspatial committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    14bd709 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2024

  1. Add additional codec support structures for imagecodecs and numcodecs.

    @sharkinsspatial
    sharkinsspatial committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    acdf0d7 View commit details
    Browse the repository at this point in the history

  2. Add codec config test for Zstd.

    @sharkinsspatial
    sharkinsspatial committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    4ba323a View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2024

  1. Include initial cf decoding tests.

    @sharkinsspatial
    sharkinsspatial committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    e14e53b View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into hdf5_reader

    @sharkinsspatial
    sharkinsspatial committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    b808ded View commit details
    Browse the repository at this point in the history

  3. Revert typo for scale_factor retrieval.

    @sharkinsspatial
    sharkinsspatial committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    b052f8c View commit details
    Browse the repository at this point in the history

  4. Update reader to use new numpy manifest representation.

    @sharkinsspatial
    sharkinsspatial committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    01a3980 View commit details
    Browse the repository at this point in the history

Commits on Jun 22, 2024

  1. Temporarily skip test until blosc netcdf4 issue is solved.

    @sharkinsspatial
    sharkinsspatial committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    c37d9e5 View commit details
    Browse the repository at this point in the history

  2. Fix Pydantic 2 migration warnings.

    @sharkinsspatial
    sharkinsspatial committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    17b30d4 View commit details
    Browse the repository at this point in the history

  3. Include hdf5plugin and imagecodecs-numcodecs in mamba test environment.

    @sharkinsspatial
    sharkinsspatial committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    f6b596a View commit details
    Browse the repository at this point in the history

  4. Mamba attempt with imagecodecs rather than imagecodecs-numcodecs.

    @sharkinsspatial
    sharkinsspatial committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    eb6e24d View commit details
    Browse the repository at this point in the history

  5. Mamba attempt with latest imagecodecs release.

    @sharkinsspatial
    sharkinsspatial committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    c85bd16 View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2024

  1. Use correct iter_chunks callback function signtature.

    @sharkinsspatial
    sharkinsspatial committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    ca435da View commit details
    Browse the repository at this point in the history

  2. Include pip based imagecodecs-numcodecs until conda-forge availability.

    @sharkinsspatial
    sharkinsspatial committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    3017951 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into hdf5_reader

    @sharkinsspatial
    sharkinsspatial committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    ccf0b73 View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2024

  1. Handle non-coordinate dims which are serialized to hdf as empty dataset.

    @sharkinsspatial
    sharkinsspatial committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    32ba135 View commit details
    Browse the repository at this point in the history

  2. Use reader_options for filetype check and update failing kerchunk call.

    @sharkinsspatial
    sharkinsspatial committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    64f446c View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into hdf5_reader

    @sharkinsspatial
    sharkinsspatial committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    1c590bb View commit details
    Browse the repository at this point in the history

Commits on Jun 30, 2024

  1. Fix chunkmanifest shaping for chunked datasets.

    @sharkinsspatial
    sharkinsspatial committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    9797346 View commit details
    Browse the repository at this point in the history

  2. Handle scale_factor attribute serialization for compressed files.

    @sharkinsspatial
    sharkinsspatial committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    c833e19 View commit details
    Browse the repository at this point in the history

  3. Include chunked roundtrip fixture.

    @sharkinsspatial
    sharkinsspatial committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    701bcfa View commit details
    Browse the repository at this point in the history

  4. Standardize xarray integration tests for hdf filters.

    @sharkinsspatial
    sharkinsspatial committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    08c988e View commit details
    Browse the repository at this point in the history

  5. Merge branch 'hdf5_reader' of https://github.com/TomNicholas/Virtuali… 

    …Zarr into hdf5_reader
    @sharkinsspatial
    sharkinsspatial committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    e6076bd View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' into hdf5_reader

    @sharkinsspatial
    sharkinsspatial committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    d684a84 View commit details
    Browse the repository at this point in the history

  7. Update reader selection logic for new filetype determination.

    @sharkinsspatial
    sharkinsspatial committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    4cb4bac View commit details
    Browse the repository at this point in the history

  8. Use decode_times for integration test.

    @sharkinsspatial
    sharkinsspatial committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    d352104 View commit details
    Browse the repository at this point in the history

  9. Standardize fixture names for hdf5 vs netcdf4 file types.

    @sharkinsspatial
    sharkinsspatial committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    3d89ea4 View commit details
    Browse the repository at this point in the history

Commits on Jul 1, 2024

  1. Handle array add_offset property for compressed data.

    @sharkinsspatial
    sharkinsspatial committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    c9dd0d9 View commit details
    Browse the repository at this point in the history

  2. Include h5py shuffle filter.

    @sharkinsspatial
    sharkinsspatial committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    db5b421 View commit details
    Browse the repository at this point in the history

  3. Make ScaleAndOffset codec last in filters list.

    @sharkinsspatial
    sharkinsspatial committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    9a1da32 View commit details
    Browse the repository at this point in the history

Commits on Jul 2, 2024

  1. Apply ScaleAndOffset codec to _FillValue since it's value is now down… 

    …stream.
    @sharkinsspatial
    sharkinsspatial committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    9b2b0f8 View commit details
    Browse the repository at this point in the history

  2. Coerce scale and add_offset values to native float for JSON serializa… 

    …tion.
    @sharkinsspatial
    sharkinsspatial committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    9ef1362 View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2024

  1. Conformant ZarrV3 codecs

    @ghidalgo3
    ghidalgo3 committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    eb16bc1 View commit details
    Browse the repository at this point in the history

  2. Update docs

    @ghidalgo3
    ghidalgo3 committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    5f1b7f9 View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2024

  1. Update virtualizarr/zarr.py 

    Co-authored-by: Tom Augspurger <[email protected]>
    @ghidalgo3 @TomAugspurger
    ghidalgo3 and TomAugspurger committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    519d45d View commit details
    Browse the repository at this point in the history

  2. Update virtualizarr/zarr.py 

    Co-authored-by: Tom Augspurger <[email protected]>
    @ghidalgo3 @TomAugspurger
    ghidalgo3 and TomAugspurger committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    76e9c8e View commit details
    Browse the repository at this point in the history

  3. Change default_fill to 0s

    @ghidalgo3
    ghidalgo3 committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    000c520 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'guhidalgo/fixmetadatacodecs' of https://github.com/ghid… 

    …algo3/VirtualiZarr into guhidalgo/fixmetadatacodecs
    @ghidalgo3
    ghidalgo3 committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    25d04b9 View commit details
    Browse the repository at this point in the history

  5. Generate permutation

    @ghidalgo3
    ghidalgo3 committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    c2e7279 View commit details
    Browse the repository at this point in the history

  6. Pythonic isinstance check

    @ghidalgo3
    ghidalgo3 committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    145960a View commit details
    Browse the repository at this point in the history

  7. Add return type to isconfigurable 

    Co-authored-by: Tom Augspurger <[email protected]>
    @ghidalgo3 @TomAugspurger
    ghidalgo3 and TomAugspurger committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    c051f04 View commit details
    Browse the repository at this point in the history

  8. Merge remote-tracking branch 'upstream/hdf5_reader' into codecs

    Tria McNeely committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    7a65fbd View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2024

  1. Changes from pair programming for zarrv3 to kerchunk file reading

    Tria McNeely committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    7b09324 View commit details
    Browse the repository at this point in the history

  2. Revert "Merge remote-tracking branch 'upstream/hdf5_reader' into codecs" 

    This reverts commit 7a65fbd, reversing
changes made to c051f04.
    Tria McNeely committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    2c59256 View commit details
    Browse the repository at this point in the history

  3. Fix unit tests

    @ghidalgo3
    ghidalgo3 committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    50c3dcd View commit details
    Browse the repository at this point in the history

Commits on Jul 22, 2024

  1. PR comments

    @ghidalgo3
    ghidalgo3 committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    ab97e63 View commit details
    Browse the repository at this point in the history

  2. Remove kwarg in dict default

    @ghidalgo3
    ghidalgo3 committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    0be0728 View commit details
    Browse the repository at this point in the history