Updates for handling CrIS SWIR data

[erinjones2]

Description

These updates have been made to enhance the processing of CrIS SWIR observations:

• Allows for CrIS SWIR channels to be used in cloud detection for CrIS SWIR observations (while keeping current functionality to use CrIS LWIR channels for CrIS LWIR and MWIR cloud detection)
• Modifies QC for CrIS SWIR observations to replace a surface check for a sun glint check
• Adds functionality for the computation and use of a scene-dependent observation error
• Adds an additional low-peaking channel (an assimilated CrIS MWIR channel) to the CrIS reader that can be used as a reference if the hard-coded LWIR channel is not present

Fixes #546

Type of change

• New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Experiments were run with these updates on Orion (and with earlier implementations of the updates on Hera). A summary of GSI enhancements and experiment results can be found here -
GSImeeting_20230509_ejones.pptx

These changes should not impact results if CrIS SWIR channels are not actively assimilated.

Regression tests were performed on Hera and can be found in /scratch2/NESDIS/nesdis-rdo1/Erin.Jones/regression_testing/tests_20230627
Results:

1/9 Test #7: [=[rrfs_3denvar_glbens]=] ........   Passed  610.22 sec
2/9 Test #9: [=[global_enkf]=] ................   Passed  688.34 sec
3/9 Test #8: [=[netcdf_fv3_regional]=] ........***Failed  730.33 sec
4/9 Test #4: [=[hwrf_nmm_d2]=] ................   Passed  790.70 sec
5/9 Test #5: [=[hwrf_nmm_d3]=] ................***Failed  799.85 sec
6/9 Test #6: [=[rtma]=] .......................   Passed  1452.57 sec
7/9 Test #3: [=[global_4denvar]=] .............   Passed  1566.73 sec
8/9 Test #2: [=[global_4dvar]=] ...............   Passed  1743.06 sec
9/9 Test #1: [=[global_3dvar]=] ...............   Passed  1986.95 sec

netcdf_fv3_regional failed a scalability test:

The runtime for netcdf_fv3_regional_loproc_updat is 79.441916 seconds and is within the maximum allowable operational time of 1200 seconds,
continuing with regression test.

The runtime for netcdf_fv3_regional_loproc_updat is 79.441916 seconds and is within the allowable threshold time of 101.347612 seconds,
continuing with regression test.

The runtime for netcdf_fv3_regional_hiproc_updat is 76.618278 seconds and is within the allowable threshold time of 87.999816 seconds,
continuing with regression test.

The memory for netcdf_fv3_regional_loproc_updat is 172372 KBs and is within the maximum allowable hardware memory limit of 2516582 KBs,
continuing with regression test.

The memory for netcdf_fv3_regional_loproc_updat is 172372 KBs and is within the maximum allowable memory of 187369 KBs,
continuing with regression test.

The results (penalty) between the two runs (netcdf_fv3_regional_loproc_updat and netcdf_fv3_regional_loproc_contrl) are reproducible.

The results (penalty) between the two runs (netcdf_fv3_regional_loproc_updat and netcdf_fv3_regional_hiproc_updat) are reproducible

The case has Failed the scalability test.
The slope for the update (3.529547 seconds per node) is less than that for the control (10.678237 seconds per node).

hwrf_nmm_d3 failed a time threshold test for hwrf_nmm_d3_loproc_updat (hwrf_nmm_d3_hiproc_updat did not fail):

The runtime for hwrf_nmm_d3_loproc_updat is 84.857470 seconds and is within the maximum allowable operational time of 1200 seconds,
continuing with regression test.

The runtime for hwrf_nmm_d3_loproc_updat is 84.857470 seconds.  This has exceeded maximum allowable threshold time of 80.168460 seconds,
resulting in Failure time-thresh of the regression test.

The runtime for hwrf_nmm_d3_hiproc_updat is 60.152334 seconds and is within the allowable threshold time of 85.971402 seconds,
continuing with regression test.

The memory for hwrf_nmm_d3_loproc_updat is 576848 KBs and is within the maximum allowable hardware memory limit of 2516582 KBs,
continuing with regression test.

The memory for hwrf_nmm_d3_loproc_updat is 576848 KBs and is within the maximum allowable memory of 635412 KBs,
continuing with regression test.

The results (penalty) between the two runs (hwrf_nmm_d3_loproc_updat and hwrf_nmm_d3_loproc_contrl) are reproducible.

The results between the two runs (hwrf_nmm_d3_loproc_updat and hwrf_nmm_d3_loproc_contrl) are reproducible
since the corresponding results are identical.

The results (penalty) between the two runs (hwrf_nmm_d3_loproc_updat and hwrf_nmm_d3_hiproc_updat) are reproducible

The results between the two runs (hwrf_nmm_d3_loproc_updat and hwrf_nmm_d3_hiproc_updat) are reproducible
since the corresponding results are identical.

The case has passed the scalability regression test.
The slope for the update (37.057704 seconds per node) is greater than or equal to that for the control (-4.835785 seconds per node).

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• New and existing tests pass with my changes
• Any dependent changes have been merged and published

DUE DATE for this PR is 8/10/2023. If this PR is not merged into develop by this date, the PR will be closed and returned to the developer.

[RussTreadon-NOAA]

@erinjones2 , please assign two peer reviewers to look over the changes in this PR. This is part of step 4 under GSI: How to Make Changes.

[erinjones2]

@RussTreadon-NOAA, @ADCollard and @emilyhcliu have agreed to review this PR; it doesn't look like I have permissions to assign them as reviewers.

[RussTreadon-NOAA]

Thank you @erinjones2 for reaching out to two peer reviewers. @ADCollard and @emilyhcliu have been added as reviewers to this PR.

[ADCollard]

I am not at all sure what the physical rationale is for doing separate longwave and shortwave cloud detections. The cloud detection scheme is not even physically appropriate for channels affected by sunlight.

If there is a good reason, we should also consider the most flexible way of implementing this so it is appropriate for all instruments.

[ADCollard]

Is there a reason for this if (cris) logic? Wouldn't the sun-glint test be applicable to all ir sounders?

[erinjones2]

It should be. We were focused only on looking at and making modifications for CrIS SW, hence the if (cris) logic. I can modify the code to make this generic for all SW, if that's preferred.

[ADCollard]

Why would cris_sw be true for GOES? Or do you want to generalise this for all IR instruments?

[erinjones2]

cris_sw shouldn't be true for GOES; this is a redundant check because I'm paranoid. It can be removed if preferred.

[ADCollard]

Why does the SW changes affect the QC decisions over high topography?

[erinjones2]

It doesn't; this is included so obs aren't counted more than once, since qc_irsnd is called twice in setuprad (calling qc_irsnd twice pertains to the LW vs SW cloud detection... see cloud detection comment)

[erinjones2]

@ADCollard The cloud detection in our experiments was done using SW for a couple of reasons: First, we were wanting to use CrIS as a proxy for a sensor without a LW band; second, we were wanting to test whether the cloud detection scheme could be done using SW channels and still produce reasonable results. It seems as if it does, so we kept it that way:

We also end up with more CrIS SW points passing QC when we use SW for cloud detection, which we attributed to the SW's higher vertical resolution. The LW channels can certainly be used instead, and the code can be modified accordingly if desired (this would also remove an extra call to qc_irsnd, but remove more SW obs in QC). Alternatively, the code can probably be modified to only use the SW channels for cloud detection if the LW channels aren't available... just let me know.

[erinjones2]

@ADCollard @emilyhcliu Getting back to this now that SNPP may be switching to MW+SW. The changes that we discussed in late June have been made in my fork; the fork will likely need to be updated now to be in line with the main develop branch again. Will any other changes be required wrt the SW updates?

[RussTreadon-NOAA]

For the time being close this PR with the possibility of this PR being reopened at a later date or a new follow-on PR being created.