Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add subType for OPERA image files #76

Merged
merged 4 commits into from
May 29, 2024
Merged

add subType for OPERA image files #76

merged 4 commits into from
May 29, 2024

Conversation

yenes56
Copy link
Contributor

@yenes56 yenes56 commented May 11, 2024

Ticket: [PODAAC-6181]

Description

Add subType to a certain image file under RelatedUrls
This PR works together with cumulus-deploy-tf change which add TaskConfig.relatedUrlSubTypeMap to support OPERA
BROWSE IMAGE SOURCE subtye.

https://github.jpl.nasa.gov/podaac/cumulus-deploy-tf/pull/347

Overview of work done

The approach is to add the following structure under collection configuration's meta
"relatedUrlSubTypeMap": [
{
"regex": "^._B01_WTR\.tif$",
"subType": "BROWSE IMAGE SOURCE"
},
{
"regex": "^.
_B02_BWTR\.tif$",
"subType": "IceBridge Portal"
}
The code logic is to assign ummg.RelatedUrls.RelatedUrl.subtype value based on whatever input.payload.granules[0].files[] items
matching the expression.

Note for Code Reviewer.

  • Due to certain subType enum values only supported by UMMG schema 1.6.6, we have to upgrade from ummg schema 1.6.5 -> 1.6.6. Hence , re-generated json schema to pojo files
  • Introduce TaskConfigFactory and TaskConfigBO which triggers some unit test code change. gradle test have passed all unit test.
  • Reviewer might see a lot of previously checked-in UMMG files been converted from one-line format to well-formatted, well-indent json. Because all the pre-stored, for-comparison-purpose ummg json have to be changed for the following reason
    ** the field representing ummg version has to change : Ex : https://github.com/podaac/cumulus-metadata-aggregator/blob/feature/PODAAC-6181/src/test/resources/ummgResults/additionalAttributes/OPERA_L3_DSWx_HLS_T14RNV_20210906T170251Z_20221026T184342Z_L8_30_v0.0.json#L2-L5
    ** Since MetadataFilesToEcho.java will return "clean up json" due to this line was added in current branch : createJson()
    all the pre-stored umm.json should have a make up by removing any item with empty array, or null value. So, storing the well fomatted json seems to be a better choice for future debugging.

Overview of verification done

  • Due to introducing TaskConfigFactory, isoXMLSpatial and subType HashArray are encapsulated within TaskBO. Retested isoXMLSpatial function by using DUMMY_SWOT_L2_HR_PIXC_1.1 collection by setting to bbx only and footprint + bbx
  • Ingest a MODIS_A-JPL 2019 granule which is not related to isoXMLSpatial nor subType.
  • Ingest the OPERA granule as below cnm request.
    Investigate the the UMMG through EarthdataClient search and found the image files matching above configured
    subType regex all set to the correct subtype.

{
"identifier": "OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4",
"collection": "OPERA_L3_DSWX-S1_PROVISIONAL_V0",
"version": "1.6.1",
"submissionTime": "2024-05-02T00:19:53.862065Z",
"product": {
"name": "OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4",
"files": [
{
"name": "OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4_BROWSE.tif",
"type": "browse",
"uri": "s3://podaac-dev-tst/OPERA_L3_DSWX-S1_PROVISIONAL_V0/OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4/OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4_BROWSE.tif",
"size": 86231,
"checksum": "4aff3db9e4aeedddc8b4ca8201ac815b",
"checksumType": "md5"
},
{
"name": "OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4_BROWSE.png",
"type": "browse",
"uri": "s3://podaac-dev-tst/OPERA_L3_DSWX-S1_PROVISIONAL_V0/OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4/OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4_BROWSE.png",
"size": 8003,
"checksum": "6211893f67440111f8fc7c5b822c63ab",
"checksumType": "md5"
},
{
"name": "OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4_B01_WTR.tif",
"type": "data",
"uri": "s3://podaac-dev-tst/OPERA_L3_DSWX-S1_PROVISIONAL_V0/OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4/OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4_B01_WTR.tif",
"size": 91275,
"checksum": "5fc5b8a7e4afeaabbc2d9065dafdfd57",
"checksumType": "md5"
},
{
"name": "OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4_B02_BWTR.tif",
"type": "data",
"uri": "s3://podaac-dev-tst/OPERA_L3_DSWX-S1_PROVISIONAL_V0/OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4/OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4_B02_BWTR.tif",
"size": 91283,
"checksum": "0cc7ecb1687e78f76d6c9a3e4069ae6c",
"checksumType": "md5"
},
{
"name": "OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4_B03_CONF.tif",
"type": "data",
"uri": "s3://podaac-dev-tst/OPERA_L3_DSWX-S1_PROVISIONAL_V0/OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4/OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4_B03_CONF.tif",
"size": 181141,
"checksum": "4a14218136b52438ef40cf17753db643",
"checksumType": "md5"
},
{
"name": "OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4_B04_DIAG.tif",
"type": "data",
"uri": "s3://podaac-dev-tst/OPERA_L3_DSWX-S1_PROVISIONAL_V0/OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4/OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4_B04_DIAG.tif",
"size": 3088750,
"checksum": "68e274d406084d0054f99ec265e2d3df",
"checksumType": "md5"
},
{
"name": "OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4.iso.xml",
"type": "metadata",
"uri": "s3://podaac-dev-tst/OPERA_L3_DSWX-S1_PROVISIONAL_V0/OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4/OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4.iso.xml",
"size": 333642,
"checksum": "ad9f90aa56d12e64da11f4ccbfc0f309",
"checksumType": "md5"
},
{
"name": "OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4.iso.xml.md5",
"type": "metadata",
"uri": "s3://podaac-dev-tst/OPERA_L3_DSWX-S1_PROVISIONAL_V0/OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4/OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4.iso.xml.md5",
"size": 32,
"checksum": "56014ccd9783565ccfe3eb22daecdeb7",
"checksumType": "md5"
},
{
"name": "OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4_B01_WTR.tif.md5",
"type": "metadata",
"uri": "s3://podaac-dev-tst/OPERA_L3_DSWX-S1_PROVISIONAL_V0/OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4/OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4_B01_WTR.tif.md5",
"size": 32,
"checksum": "81abc30194d7c8b7f886fb9ca3c45350",
"checksumType": "md5"
},
{
"name": "OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4_B02_BWTR.tif.md5",
"type": "metadata",
"uri": "s3://podaac-dev-tst/OPERA_L3_DSWX-S1_PROVISIONAL_V0/OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4/OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4_B02_BWTR.tif.md5",
"size": 32,
"checksum": "59693f63f14c4e17c966218f5fbacb14",
"checksumType": "md5"
},
{
"name": "OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4_B03_CONF.tif.md5",
"type": "metadata",
"uri": "s3://podaac-dev-tst/OPERA_L3_DSWX-S1_PROVISIONAL_V0/OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4/OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4_B03_CONF.tif.md5",
"size": 32,
"checksum": "d7f363fa88ae16f90e7a4ac5c8a7ebee",
"checksumType": "md5"
},
{
"name": "OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4_B04_DIAG.tif.md5",
"type": "metadata",
"uri": "s3://podaac-dev-tst/OPERA_L3_DSWX-S1_PROVISIONAL_V0/OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4/OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4_B04_DIAG.tif.md5",
"size": 32,
"checksum": "1d3013e414bf2cb8f1afa299071738bb",
"checksumType": "md5"
},
{
"name": "OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4_BROWSE.png.md5",
"type": "metadata",
"uri": "s3://podaac-dev-tst/OPERA_L3_DSWX-S1_PROVISIONAL_V0/OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4/OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4_BROWSE.png.md5",
"size": 32,
"checksum": "a86797cda1092900a07f697e5bc286d6",
"checksumType": "md5"
},
{
"name": "OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4_BROWSE.tif.md5",
"type": "metadata",
"uri": "s3://podaac-dev-tst/OPERA_L3_DSWX-S1_PROVISIONAL_V0/OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4/OPERA_L3_DSWx-S1_T35TPJ_20240214T155240Z_20240502T001754Z_S1A_30_v0.4_BROWSE.tif.md5",
"size": 32,
"checksum": "2e31b0fcdd21aafc4c3b636b2f989a8e",
"checksumType": "md5"
}
],
"dataVersion": "0.0"
},
"provider": "JPL-OPERA",
"trace": "opera-int-fwd"
}

Tested in SIT:

Explain how you tested in SIT, if applicable

PR checklist:

  • Linted
  • Unit tests
  • Addressed Snyk vulnerabilities
  • Updated changelog
  • Tested in SIT
  • Documentation / User-Guide Updated

See Pull Request Review Checklist for pointers on reviewing this pull request

@yenes56 yenes56 reopened this May 15, 2024
@skorper skorper merged commit 99698f0 into develop May 29, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants