Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes #1618. Add zstandard support #3056

Open
wants to merge 5 commits into
base: develop
Choose a base branch
from

Conversation

mathomp4
Copy link
Member

@mathomp4 mathomp4 commented Sep 25, 2024

Types of change(s)

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Trivial change (affects only documentation or cleanup)
  • Refactor (no functional changes, no api changes)

Checklist

  • Tested this change with a run of GEOSgcm
  • Ran the Unit Tests (make tests)

Description

This PR adds support for zstandard compression in History via zstandard_level.

Much like netCDF quantize support, if netCDF-C wasn't built with zstandard support, then MAPL will not allow a user to set zstandard_support. We also prevent a user from setting both deflate and zstandard_level at the same time. Also, two new ExtData tests are added.

NOTE: At the moment, this support is only available via Spack builds of MAPL (and GEOS). The reason is that zstandard support in netCDF-C is only available via shared library (because it uses HDF5 plugins). For historic reasons, ESMA Baselibs uses static HDF5, netCDF, etc.

cc: @edwardhartnett

Related Issue

Closes #1618

@mathomp4 mathomp4 added the 0 Diff The changes in this pull request have verified to be zero-diff with the target branch. label Sep 25, 2024
@mathomp4 mathomp4 self-assigned this Sep 25, 2024
@mathomp4
Copy link
Member Author

mathomp4 commented Sep 25, 2024

Yup. This is currently not working. I've done something wrong, but not sure what yet. Turns out, enabling zstandard doesn't actually enable it! Bug hunting time.

ETA: Bug is found! Huzzah!

@mathomp4 mathomp4 marked this pull request as ready for review September 26, 2024 22:09
@mathomp4 mathomp4 requested a review from a team as a code owner September 26, 2024 22:09
@mathomp4 mathomp4 changed the title WIP: Fixes #1618. Add zstandard support Fixes #1618. Add zstandard support Sep 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0 Diff The changes in this pull request have verified to be zero-diff with the target branch.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant