SWOT collections failing temporal subsetting

[jamesfwood]

The SWOT collections are failing temporal subsetting.

This simulated harmony request works, but the real collections fail.

Simulated (Working):
https://harmony.earthdata.nasa.gov/C2152044763-POCLOUD/ogc-api-coverages/1.0.0/collections/ssh_karin%2Cssha_karin%2Cssh_karin_2%2Cssha_karin_2%2Ctime%2Ctime_tai%2Clongitude%2Clatitude/coverage/rangeset?subset=lon(-125.57812%3A-124.73438)&subset=lat(34.84931%3A35.83342)&subset=time(%222012-01-01T00%3A00%3A00Z%22%3A%222012-03-31T23%3A59%3A59Z%22)&format=application%2Fnetcdf

Make all the real collections work also:

SWOT_L2_LR_SSH_BASIC_1.0
SWOT_L2_LR_SSH_EXPERT_1.0
SWOT_L2_LR_SSH_UNSMOOTHED_1.0
SWOT_L2_LR_SSH_WINDWAVE_1.0
SWOT_L2_NALT_IGDR_SSHA_1.0
SWOT_L2_NALT_OGDR_SSHA_1.0

[jamesfwood]

References here:
https://wiki.jpl.nasa.gov/display/PD/Data+Subscriber+1.15.0+UAT

[sliu008]

#190

[jamesfwood]

Test in Data Subscriber when ready.

[jamesfwood]

@davidcolemanjpl l2ss-py was updated in harmony in UAT only. Please try to test this in UAT only. You may need help from @jjmcnelis to setup the test case. Thanks!

[davidcolemanjpl]

@jamesfwood I'l review, but will need more details regarding this validation:

E.g.,
Inputs / Expected results

Version(s) under test (l2ss-py) --
'l2ss-py:2.6.0rc11'

[jamesfwood]

@davidcolemanjpl Please test using the Data Subscriber. It's only deployed to UAT so far, so test in UAT. I think you can get a test plan from either @jjmcnelis or @skorper. This was an important fix for them.
Let me know if you need more help. Thanks!

[jamesfwood]

@davidcolemanjpl you can also test this using the harmony call directly:

https://harmony.uat.earthdata.nasa.gov/C1256507988-POCLOUD/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?forceAsync=true&subset=lat(34.84931:35.83342)&subset=lon(-125.57812:-124.73438)&subset=time("2023-04-18T18:00:00%2B00:00":"2023-04-20T07:00:00%2B00:00")&skipPreview=true

Something like this, but update the subset lon and lat and tweak the time a bit.

[jamesfwood]

@davidcolemanjpl here's another one that should work. Just tweak the times.

https://harmony.uat.earthdata.nasa.gov/C1256507988-POCLOUD/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?forceAsync=true&subset=time("2023-04-19T02:00:00%2B00:00":"2023-04-19T04:00:00%2B00:00")&skipPreview=true

Hopefully you can test this soon so we can do a release this week. Thanks!

[davidcolemanjpl]

l2ss-py:2.6.0rc14; podaac-data-subscriber v1.15.0a3
UAT

@jamesfwood
Thanks, ...the harmony jobID seems to trigger as expected when using call for the SWOT_L2_LR_SSH_BASIC_1.0 collection.

Are there example, valid harmony calls for exercising other SWOT sub-collections under test?:
SWOT_L2_LR_SSH_EXPERT_1.0
SWOT_L2_LR_SSH_UNSMOOTHED_1.0
SWOT_L2_LR_SSH_WINDWAVE_1.0
SWOT_L2_NALT_IGDR_SSHA_1.0
SWOT_L2_NALT_OGDR_SSHA_1.0

note: When attempting to execute a harmony call directly using the SWOT_L2_LR_SSH_EXPERT_1.0 collection (C1256122852-POCLOUD), the jobID hung at the running status
See jobID : efe48442-72ee-454f-8e13-93c3202a726e
https://harmony.uat.earthdata.nasa.gov/workflow-ui/efe48442-72ee-454f-8e13-93c3202a726e

See jobID: 6d36b26c-be0e-4423-840c-a005fa0dc903. - FAILED
WorkItem failed: podaac/l2ss-py:2.6.0rc14: Service request failed with an unknown error

https://harmony.uat.earthdata.nasa.gov/C1256122852-POCLOUD/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?forceAsync=true&subset=lat(34.84931:35.83342)&subset=lon(-125.57812:-124.73438)&subset=time("2023-04-17T18:00:00%2B00:00":"2023-04-19T07:00:00%2B00:00")&skipPreview=true

See jobID: 02e0e68f-bf23-4960-b0e0-4ce0687672da
"The job failed with 2 errors. See the errors field for more details"

note2: The L2SS-Py functionality seems to work okay when sub-setting SWOT_L2_LR_SSH_EXPERT_1.0 collection nominally (via HiTIDE) :
Can see the successful jobID 14682326-2b58-4b85-9d1e-097c3a928653
https://harmony.uat.earthdata.nasa.gov/jobs/14682326-2b58-4b85-9d1e-097c3a928653

podaac-data-subscriber v1.15.0a3: - anomaly
SWOT_L2_LR_SSH_BASIC_1.0

% podaac-data-downloader -c SWOT_L2_LR_SSH_BASIC_1.0 -d ./data --start-date 2023-04-17T18:00:00Z --end-date 2023-04-21T07:00:00Z --subset -b="-125.57812,34.84931,-124.73438,35.83342"
[2023-10-04 12:55:57,631] {podaac_data_downloader.py:382} ERROR - Uncaught exception occurred during execution.
Traceback (most recent call last):
File "/Users/colemand/work/env/lib/python3.10/site-packages/subscriber/podaac_data_downloader.py", line 380, in main
run()
File "/Users/colemand/work/env/lib/python3.10/site-packages/subscriber/podaac_data_downloader.py", line 142, in run
collection_id = pa.get_cmr_collection_id(
File "/Users/colemand/work/env/lib/python3.10/site-packages/subscriber/podaac_access.py", line 574, in get_cmr_collection_id
raise ValueError(f'No collections found in CMR for {collection_short_name}/{provider}')
ValueError: No collections found in CMR for SWOT_L2_LR_SSH_BASIC_1.0/POCLOUD

(I'll keep investigating this)

[davidcolemanjpl]

l2ss-py:2.6.0rc14; podaac-data-subscriber v1.15.0a3
UAT
SWOT_L2_LR_SSH_WINDWAVE_1.0 (C1256507990-POCLOUD)

See jobID: c3b76190-f155-4f5e-aa8a-bc2d5da6270e - FAILED
harmony call :
https://harmony.uat.earthdata.nasa.gov/C1256507990-POCLOUD/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?forceAsync=true&subset=lat(34.84931:35.83342)&subset=lon(-125.57812:-124.73438)&subset=time("2023-04-18T18:00:00%2B00:00":"2023-04-20T07:00:00%2B00:00")&skipPreview=true

"The job failed with 3 errors. See the errors field for more details"

https://harmony.uat.earthdata.nasa.gov/jobs/c3b76190-f155-4f5e-aa8a-bc2d5da6270e

podaac-data-subscriber v1.15.0a3: - anomaly:
"ValueError: No collections found in CMR for SWOT_L2_LR_SSH_WINDWAVE_1.0/POCLOUD"

note:
The L2SS-Py functionality seems to work okay when subsetting SWOT_L2_LR_SSH_WINDWAVE collection nominally (via HiTIDE)

[davidcolemanjpl]

l2ss-py:2.6.0rc14;
podaac-data-subscriber v1.15.0a3

UAT

SWOT_L2_LR_SSH_BASIC_1.0:
Data subscriber issue: ValueError: No collections found in CMR for C1256507988-POCLOUD/POCLOUD

SWOT_L2_LR_SSH_EXPERT_1.0:
Data subscriber issue: ValueError: No collections found in CMR for SWOT_L2_LR_SSH_EXPERT_1.0/POCLOUD

When using Harmony call directly, the subset failed for SWOT_L2_LR_SSH_EXPERT_1.0 (see Harmony jobID: 178ea40f-643a-4d78-8ea3-c8e8c49ac038)
"WorkItem failed: podaac/l2ss-py:2.6.0rc14: Service request failed with an unknown error"
https://harmony.uat.earthdata.nasa.gov/jobs/178ea40f-643a-4d78-8ea3-c8e8c49ac038

SWOT_L2_LR_SSH_UNSMOOTHED_1.0:
Data subscriber issue: ValueError: No collections found in CMR for SWOT_L2_LR_SSH_UNSMOOTHED_1.0/POCLOUD

SWOT_L2_LR_SSH_WINDWAVE_1.0:
Data subscriber issue: ValueError: No collections found in CMR for SWOT_L2_LR_SSH_WINDWAVE_1.0/POCLOUD

When using Harmony call Directly, the subset FAILED for SWOT_L2_LR_SSH_WINDWAVE_1.0
(see Failed jobID: c3b76190-f155-4f5e-aa8a-bc2d5da6270e)
"WorkItem failed: podaac/l2ss-py:2.6.0rc14: Service request failed with an unknown error"
https://harmony.uat.earthdata.nasa.gov/jobs/c3b76190-f155-4f5e-aa8a-bc2d5da6270e

SWOT_L2_NALT_IGDR_SSHA_1.0:
l2ss-py:2.6.0rc14 - OKAY - PASS
podaac-data-subscriber v1.15.0a3 - OKAY - PASS

SWOT_L2_NALT_OGDR_SSHA_1.0:
l2ss-py:2.6.0rc14 - OKAY - PASS
podaac-data-subscriber v1.15.0a3 - OKAY - PASS

can also see related podaac-data-subscriber ticket:
PODAAC-5859

[jamesfwood]

@davidcolemanjpl how do you run data subscriber with UAT sources? collection ids, using uat services, etc.

[davidcolemanjpl]

Correct, utilized the respective Data Subscriber UAT Install Instructions

[jamesfwood]

@davidcolemanjpl We deployed a new l2ss-py to OPS. Can you please test this again in OPS?

[jamesfwood]

Harmony may be down right now, so please test when it's back running again.
Test here:
https://harmony.earthdata.nasa.gov/workflow-ui

[davidcolemanjpl]

HITIDE UI OPS V4.16.1
L2SS-PY:v2.6.0
CONCISE: v0.6.1

@jamesfwood

Related SWOT L2 collections currently displaying in HiTIDE UI (OPS):
SWOT_L2_NALT_IGDR_SSHA_1.0 -- (C2628598397-POCLOUD) -- PASSED

https://harmony.earthdata.nasa.gov/jobs/5945fd10-8785-4e56-9d7f-032beccc6623 (L2SS-py PASS via HiTIDE UI)
https://harmony.earthdata.nasa.gov/jobs/a6feb324-8b34-4900-b776-f7eebc1a0ae6 (L2SS-py PASS using Harmony call)

SWOT_L2_NALT_OGDR_SSHA_1.0 -- (C2628593693-POCLOUD) -- PASSED
https://harmony.earthdata.nasa.gov/jobs/5df9ba70-5243-456a-b7b2-d00a1ee5adcd (L2SS-py PASS via HiTIDE UI)
https://harmony.earthdata.nasa.gov/jobs/413e36e1-fad7-493d-a7c6-7f988e8dbb3d (L2SS-py PASS using Harmony call)

note: the following SWOT L2 LR collections have not been implemented thus far in HiTIDE OPS UI and/or EARTHDATASEARCH:
'BLOCKED'
**SWOT_L2_LR_SSH_BASIC_1.0

SWOT_L2_LR_SSH_EXPERT_1.0

SWOT_L2_LR_SSH_UNSMOOTHED_1.0
SWOT_L2_LR_SSH_WINDWAVE_1.0**

See related issue for SWOT collections: podaac/hitide#13

---

UPDATE: 10/18:
I was able to create / utilze Harmony call in OPS for SWOT_L2_LR_SSH_EXPERT_1.0 granules : (L2SS-py - Conditional PASS using Harmony call):
(SWOT Simulated Level-2 KaRIn SSH from MITgcm ECCO LLC4320 for Science Version 1 collection / SWOT_SIMULATED_L2_KARIN_SSH_ECCO_LLC4320_SCIENCE_V1 / C2152044763-POCLOUD)

https://harmony.earthdata.nasa.gov/C2152044763-POCLOUD/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?forceAsync=true&subset=lat(34.84931:35.83342)&subset=lon(-125.57812:-124.73438)&subset=time("2012-03-01T18:00:00%2B00:00":"2023-04-21T07:00:00%2B00:00")&skipPreview=true

See Harmony jobID : a98c7b20-8634-4b02-a867-f3bee4cc4c45

"The job has completed with errors. See the errors field for more details"
"message":"WorkItem failed: podaac/l2ss-py:2.6.0: Service request failed with an unknown error"

https://harmony.earthdata.nasa.gov/jobs/a98c7b20-8634-4b02-a867-f3bee4cc4c45