Release/2.5.0 (#163)

* /version 2.5.0-alpha.0 * /version 2.5.0-alpha.1 * Adding collections OPS: C2599212091-POCLOUD * /version 2.5.0-alpha.2 * Adding collections OPS: C2208421887-POCLOUD * /version 2.5.0-alpha.3 * Adding collections OPS: C2205121449-POCLOUD * /version 2.5.0-alpha.4 * update ascending flag variable clean up (#154) * update ascending flag variable clean up * Update changelog * Add sndr ascending/descending type * Update testing file * added sndr file to list files to test * change tests to call subset method * change assertion test if box returned is true * change assertion test if box returned is true * expand time test zone * Ascending flag does not coorespond to the asc_node_time * Ascending flag does not coorespond to the asc_node_time * Updated werkzeug=2.2.3 and cryptography=='39.0.1' * Update poetry * Ran poetry update * Update poetry version used in pipeline * Fix pylint errors * Bump python version used in build * rollback numpy=1.24 to numpy=1.23.5 --------- Co-authored-by: nlensse1 <[email protected]> Co-authored-by: Frank Greguska <[email protected]> * /version 2.5.0a5 * Adding collections OPS: C2274919541-POCLOUD * Adding collections UAT: C1238658392-POCLOUD C1243175554-POCLOUD * /version 2.5.0a6 * Adding collections UAT: C1240817851-POCLOUD C1238621185-POCLOUD C1241042620-POCLOUD C1240739719-POCLOUD C1256420924-POCLOUD C1245295751-POCLOUD C1238621118-POCLOUD C1238621219-POCLOUD C1240739606-POCLOUD C1242735870-POCLOUD C1240739611-POCLOUD C1240739704-POCLOUD C1238657961-POCLOUD C1238658049-POCLOUD C1241042621-POCLOUD C1238621092-POCLOUD C1238658050-POCLOUD C1240739764-POCLOUD C1238621115-POCLOUD C1238657960-POCLOUD C1256524295-POCLOUD C1245295750-POCLOUD C1240739768-POCLOUD C1238621176-POCLOUD * /version 2.5.0a7 * Adding collections UAT: C1238621178-POCLOUD C1238658088-POCLOUD C1238687534-POCLOUD C1238687284-POCLOUD C1256445396-POCLOUD C1238621172-POCLOUD C1240739654-POCLOUD C1238658086-POCLOUD C1238621186-POCLOUD C1238687546-POCLOUD C1238658080-POCLOUD C1240739688-POCLOUD C1238621182-POCLOUD * /version 2.5.0a8 * Feature/podaac 5538 (#156) * update l2ss to load each dataset variable individually to write to file * remove unused variable * remove test variable that is not used * /version 2.5.0a9 * Adding collections OPS: C2296989383-POCLOUD C2296989388-POCLOUD C2296989390-POCLOUD * /version 2.5.0a10 * Adding collections UAT: C1234208436-POCLOUD C1238570311-POCLOUD * /version 2.5.0a11 * Adding collections UAT: C1256783391-POCLOUD C1256783386-POCLOUD * /version 2.5.0a12 * Feature/podaac 5537 (#157) * update l2ss py to open hdf5 files if we fail to open as netcdf, and change way we copy attributes in case it has a / * fix pylint and removed comment * update changelog * /version 2.5.0a13 * Feature/podaac 5537 (#161) * update l2ss py to open hdf5 files if we fail to open as netcdf, and change way we copy attributes in case it has a / * fix pylint and removed comment * update changelog * change so that were able to process S1 and S2 dtypes * fix pylint and flake8 * revert spaces * remove spaces * remove spaces * /version 2.5.0a14 * Bump tornado from 6.2 to 6.3.2 (#160) Bumps [tornado](https://github.com/tornadoweb/tornado) from 6.2 to 6.3.2. - [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst) - [Commits](tornadoweb/tornado@v6.2.0...v6.3.2) --- updated-dependencies: - dependency-name: tornado dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump requests from 2.28.2 to 2.31.0 (#159) Bumps [requests](https://github.com/psf/requests) from 2.28.2 to 2.31.0. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.28.2...v2.31.0) --- updated-dependencies: - dependency-name: requests dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * /version 2.5.0a15 * Feature/issue 155 (#158) * add group variable fix test * Updated MLS fix with TEMPO * update linting * fix pylinting * Fix pylint * fix pylint * add extra blank line * Update group_vars extending function * fix linting * attempted fix with variable subsetting * pylinting updated * Update subset.py and lint * Update subset.py and lint * fix unique groups * fix unique groups * fix unique groups * fix unique groups * add unit test * remove print statements * remove whitespace * CHANGELOG updated * add space in file * remove whitespace * Don't add a variable twice * fix linting * include test for latitude in root group * include test for latitude in root group * fix group handling * fix group handling * streamline group list control flow * add missing import * remove indent, remove print statement --------- Co-authored-by: nlensse1 <[email protected]> Co-authored-by: jamesfwood <[email protected]> Co-authored-by: danielfromearth <[email protected]> * /version 2.5.0a16 * Updated changelog for release 2.5.0 * /version 2.6.0-alpha.0 * /version 2.5.0-rc.1 * trigger rebuild * /version 2.5.0rc2 * /version 2.5.0rc3 * fix version format * /version 2.5.0rc4 * Bugfix/fix docker versioning type (#164) * update docker versioning to pep440 * update changelog * /version 2.5.0rc5 * undo replacing slashes with _ in values for attributes (#165) * /version 2.5.0rc6 * Bump cryptography from 39.0.2 to 41.0.0 (#167) Bumps [cryptography](https://github.com/pyca/cryptography) from 39.0.2 to 41.0.0. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@39.0.2...41.0.0) --- updated-dependencies: - dependency-name: cryptography dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * /version 2.6.0a1 * Feature/issue 162 (#166) * add group variable fix test * Updated MLS fix with TEMPO * update linting * fix pylinting * Fix pylint * fix pylint * add extra blank line * Update group_vars extending function * fix linting * attempted fix with variable subsetting * pylinting updated * Update subset.py and lint * Update subset.py and lint * fix unique groups * fix unique groups * fix unique groups * fix unique groups * add unit test * remove print statements * remove whitespace * CHANGELOG updated * add space in file * remove whitespace * Don't add a variable twice * fix linting * add root variable subsetting * add root variable subsetting * include file ext in tests for argument * update tests * update tests * allow subsetting from one lat var name * allow subsetting from one lat var name * allow subsetting from one lat var name * allow temporal subsetting for MLS and OCO3 * updated changelog * Update test change * remove itertools extra import * Update checking the var_name --------- Co-authored-by: nlensse1 <[email protected]> * /version 2.6.0a2 * Feature/issue 168 (#169) * make separate copy of test data file to get expected results before subsetting * update CHANGELOG.md * lint the test_subset.py file via pylint * /version 2.6.0a3 * /version 2.5.0rc7 * deploy * /version 2.5.0rc8 * re-add get timevar process for MLS (#171) * re-add get timevar process for MLS * Update changelog * Add comments for get time var name * Add comments for get time var name --------- Co-authored-by: nlensse1 <[email protected]> * /version 2.6.0a4 * /version 2.5.0rc9 * Adding collections UAT: C1256507988-POCLOUD * Adding collections UAT: C1256507990-POCLOUD C1256122852-POCLOUD C1256507989-POCLOUD * /version 2.6.0a5 * Feature/issue 173 (#174) * Update xarray dataset merge * update subsetting for variables without sptaital dimensions * Update variable names and Changelog * Update variable name * Update variable name --------- Co-authored-by: nlensse1 <[email protected]> * /version 2.6.0a6 * /version 2.5.0rc10 * Adding collections UAT: C1238621112-POCLOUD * /version 2.6.0a7 * Adding collections OPS: C2601585875-POCLOUD C2628593693-POCLOUD C2601584109-POCLOUD C2601583089-POCLOUD C2601581863-POCLOUD C2628598397-POCLOUD * /version 2.6.0a8 * /version 2.5.0rc11 * trigger redeploy * /version 2.5.0rc13 --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: l2ss-py bot <[email protected]> Co-authored-by: James Wood <[email protected]> Co-authored-by: podaac-cloud-dsa <[email protected]> Co-authored-by: Nick Lenssen <[email protected]> Co-authored-by: nlensse1 <[email protected]> Co-authored-by: Frank Greguska <[email protected]> Co-authored-by: sliu008 <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: danielfromearth <[email protected]> Co-authored-by: Daniel Kaufman <[email protected]>
podaac · Jul 19, 2023 · 76c8ccd · 76c8ccd
1 parent 3f63590
commit 76c8ccd
Show file tree

Hide file tree

Showing 15 changed files with 2,097 additions and 433 deletions.
diff --git a/.github/workflows/build-pipeline.yml b/.github/workflows/build-pipeline.yml
@@ -24,11 +24,11 @@ jobs:
       - uses: actions/checkout@v2
       - uses: actions/setup-python@v2
         with:
-          python-version: 3.8
+          python-version: 3.9
       - name: Install Poetry
         uses: abatilo/[email protected]
         with:
-          poetry-version: 1.1.8
+          poetry-version: 1.3.2
       - name: Get version
         id: get-version
         run: |
@@ -61,7 +61,7 @@ jobs:
           if [ "$BUMP_RC" = true ]; then
             poetry version prerelease
           else
-            poetry version ${GITHUB_REF#refs/heads/release/}-rc.1
+            poetry version ${GITHUB_REF#refs/heads/release/}rc1
           fi
           echo "software_version=$(poetry version | awk '{print $2}')" >> $GITHUB_ENV
           echo "venue=uat" >> $GITHUB_ENV
@@ -99,7 +99,7 @@ jobs:
             -Dsonar.tests=tests/
             -Dsonar.projectName=l2ss-py
             -Dsonar.projectVersion=${{ env.software_version }}
-            -Dsonar.python.version=3.7,3.8,3.9
+            -Dsonar.python.version=3.8,3.9,3.10
       - name: Run Snyk as a blocking step
         uses: snyk/actions/python-3.8@master
         env:
@@ -208,7 +208,7 @@ jobs:
         with:
           images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
           tags: |
-            type=semver,pattern={{version}},value=${{ env.software_version }}
+            type=pep440,pattern={{version}},value=${{ env.software_version }}
             type=raw,value=${{ env.venue }}
       - name: Wait for package
         if: |

diff --git a/.github/workflows/release-created.yml b/.github/workflows/release-created.yml
@@ -25,7 +25,7 @@ jobs:
       - name: Install Poetry
         uses: abatilo/[email protected]
         with:
-          poetry-version: 1.1.8
+          poetry-version: 1.3.2
       - name: Bump minor version
         run: |
           poetry version ${GITHUB_REF#refs/heads/release/}

diff --git a/.pylintrc b/.pylintrc
@@ -484,5 +484,5 @@ min-public-methods=2
 
 # Exceptions that will emit a warning when being caught. Defaults to
 # "BaseException, Exception".
-overgeneral-exceptions=BaseException,
-                       Exception
+overgeneral-exceptions=builtins.BaseException,
+                       builtins.Exception
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -13,6 +13,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Security
 
 
+## [2.5.0]
+### Added
+### Changed
+### Deprecated 
+### Removed
+### Fixed
+- [issue/153](https://github.com/podaac/l2ss-py/issues/153): Remove the asc_node_tai93 variable when blank in the SNDR collections for xarray.decode_times to decode.
+- PODAAC-5538: Reduce memory footprint of l2ss by loading each variable individually to write to memory
+- [issue/155](https://github.com/podaac/l2ss-py/issues/155): lat var name prefix now generalized to unique group names. Group variables determined in subset_bbox using the unique group name.
+- [issue/162](https://github.com/podaac/l2ss-py/issues/162): allow time variable subsetting differences handled for MLS and OCO3 collections. At the moment: assuming 'he5' files to be tai93 files is a fine assumption. Xarray does not decode these times in he5 files
+- [issue/168](https://github.com/podaac/l2ss-py/issues/168): A separate copy of test data is used to get expected results before subsetting at the beginning of two test functions --- test_subset_empty_bbox() and test_specified_variables() --- that rely on opening the dataset more than once.
+- PODAAC-5537: Fixed AQUARIUS_L2_SSS_V5 dataset
+- Update github action docker versioning to use pep440
+- [issue/170](https://github.com/podaac/l2ss-py/issues/170): Prioritize getting time variables if 'time' == var_time_name over 'time' being in the var name such as 'SunLocalTime'.
+- [issue/173](https://github.com/podaac/l2ss-py/issues/173): Variables without any spatial dimensions in "indexers" array should get copied over to the subsetting new_dataset after the xarray.where() function is applied.
+### Security
+
 ## [2.4.0]
 ### Added
 ### Changed

diff --git a/cmr/ops_associations.txt b/cmr/ops_associations.txt
@@ -73,3 +73,16 @@ C2068529568-POCLOUD
 C2036882397-POCLOUD
 C2036882499-POCLOUD
 C2036882492-POCLOUD
+C2599212091-POCLOUD
+C2208421887-POCLOUD
+C2205121449-POCLOUD
+C2274919541-POCLOUD
+C2296989383-POCLOUD
+C2296989388-POCLOUD
+C2296989390-POCLOUD
+C2601585875-POCLOUD
+C2628593693-POCLOUD
+C2601584109-POCLOUD
+C2601583089-POCLOUD
+C2601581863-POCLOUD
+C2628598397-POCLOUD
diff --git a/cmr/uat_associations.txt b/cmr/uat_associations.txt
@@ -41,3 +41,51 @@ C1242387621-POCLOUD
 C1238658389-POCLOUD
 C1244810554-POCLOUD
 C1256420925-POCLOUD
+C1238658392-POCLOUD
+C1243175554-POCLOUD
+C1240817851-POCLOUD
+C1238621185-POCLOUD
+C1241042620-POCLOUD
+C1240739719-POCLOUD
+C1256420924-POCLOUD
+C1245295751-POCLOUD
+C1238621118-POCLOUD
+C1238621219-POCLOUD
+C1240739606-POCLOUD
+C1242735870-POCLOUD
+C1240739611-POCLOUD
+C1240739704-POCLOUD
+C1238657961-POCLOUD
+C1238658049-POCLOUD
+C1241042621-POCLOUD
+C1238621092-POCLOUD
+C1238658050-POCLOUD
+C1240739764-POCLOUD
+C1238621115-POCLOUD
+C1238657960-POCLOUD
+C1256524295-POCLOUD
+C1245295750-POCLOUD
+C1240739768-POCLOUD
+C1238621176-POCLOUD
+C1238621178-POCLOUD
+C1238658088-POCLOUD
+C1238687534-POCLOUD
+C1238687284-POCLOUD
+C1256445396-POCLOUD
+C1238621172-POCLOUD
+C1240739654-POCLOUD
+C1238658086-POCLOUD
+C1238621186-POCLOUD
+C1238687546-POCLOUD
+C1238658080-POCLOUD
+C1240739688-POCLOUD
+C1238621182-POCLOUD
+C1234208436-POCLOUD
+C1238570311-POCLOUD
+C1256783391-POCLOUD
+C1256783386-POCLOUD
+C1256507988-POCLOUD
+C1256507990-POCLOUD
+C1256122852-POCLOUD
+C1256507989-POCLOUD
+C1238621112-POCLOUD
diff --git a/podaac/subsetter/group_handling.py b/podaac/subsetter/group_handling.py
@@ -170,7 +170,11 @@ def _rename_variables(dataset: xr.Dataset, base_dataset: nc.Dataset, start_date)
                 encoded_var = cf_dt_coder.encode(dataset.variables[var_name])
                 variable = encoded_var
 
-        var_attrs = variable.attrs
+        var_attrs = {}
+        for key, value in variable.attrs.items():
+            new_key = key.replace("/", "_") if isinstance(key, str) else key
+            var_attrs[new_key] = value
+
         fill_value = var_attrs.get('_FillValue')
         var_attrs.pop('_FillValue', None)
         comp_args = {"zlib": True, "complevel": 1}
@@ -182,6 +186,8 @@ def _rename_variables(dataset: xr.Dataset, base_dataset: nc.Dataset, start_date)
             var_data = np.array(variable.data)
         elif variable.dtype == 'timedelta64[ns]':
             var_group.createVariable(new_var_name, 'i4', var_dims, fill_value=fill_value, **comp_args)
+        elif variable.dtype in ['|S1', '|S2']:
+            var_group.createVariable(new_var_name, variable.dtype, var_dims, fill_value=fill_value)
         else:
             var_group.createVariable(new_var_name, variable.dtype, var_dims, fill_value=fill_value, **comp_args)
 
@@ -190,7 +196,10 @@ def _rename_variables(dataset: xr.Dataset, base_dataset: nc.Dataset, start_date)
 
         # Copy data
         var_group.variables[new_var_name].set_auto_maskandscale(False)
-        var_group.variables[new_var_name][:] = var_data
+        if variable.dtype in ['|S1', '|S2']:
+            var_group.variables[new_var_name][:] = variable.values
+        else:
+            var_group.variables[new_var_name][:] = var_data
 
 
 def h5file_transform(finput: str) -> Tuple[nc.Dataset, bool]: