Something is amiss in handling IASI (perhaps all hyperspectral IR)

[rtodling]

As I introduce T-skin as a control variable, I am re-doing some of the exercise I've done before of looking at each of the main instruments increments and how they compare w/ GSI's. IASI (and CrIS-FSR) has always been in the suspicious list. I want to have a record of this here so we can work together to address whatever issue affects hyperspectral IR.

The temperature increment for microwave instruments such as ATMS (clear sky) and GMI (all sky) are quite consistent between GSI and JEDI. The corresponding T-skin increments are also quite consistent - with still a puzzle over Antartica (but not a show stopper - see e.g., https://github.com/JCSDA-internal/fv3-jedi/pull/1257.

For IASI, the story is quite alarmingly different. For starts, GSI does not fully converge its minimization when given up to 100 iteration to crank; but so be it. JEDI, on the other hand, seems to converge in about 40 something iterations, which is suspicious in itself. But the real issue is how different the increments from both systems look (perhaps not a surprise given the convergence differences - chicken and egg).

Interestingly enough, the increments on specific humidity between the two systems are quite reasonable - here is an example at 850 hPa - GSI (top) JEDI (bottom)

Unfortunately, this is what temperature looks like at the same level:

notice T increments are multiplied by a factor of 2 for plotting enhancement purposes; still -diff in increments are larger than increments themselves.

Now here is what T-skin increments looks like:

It looks as though the weight given to whatever induces the sensitivity of Tb to T-skin is overweighted.

[rtodling]

Here is another set of diagnostics. These are now calculated as follows:

displayed-inc-diff = aave(abs(jedi(temperature),g) - aave(abs(gsi(temperature),g)

for three instruments/platforms: GMI-GPM; ATMS-N20 and IASI-Metop-B, respectively:

GMI

ATMS-N20

IASI-MetopB

From my perspective, differences of a few hundredths of a degree is this type of measure should be ok - GMI differs in the thousandths of degrees (perhaps because its increments are considerable smaller in magnitude than those of ATMS); ATMS differences seem a little large (perhaps reasonable down in the troposphere, but not so in the stratosphere esp around 1 hPa); IASI differences seem completely unreasonable, pretty much along the entire profile.

[rtodling]

Another curious result comes from running both GSI and JEDI, for IASI-MetopB, with and without correlated errors.

Now here is what the measure above looks like for the two corresponding GSI experiments (no-Correlations-minus-Correlations):

and here is a similar results from JEDI:

In this sense, both code (GSI and JEDI) behave similarly.

These obviously don't show how different JEDI's results are from GSI's (see above), which indicate JEDI's to be the odd ball. But when it comes to on/off correlated R, the behavior of JEDI is more what I'd expect to see.

Indeed even in the differences above one can see JEDI's differences near the surface are larger than GSI's - this is clearly visible in the T-skin increment differences that qualitatively don't change much from what's been illustrate above already.

[rtodling]

I am finding out that this whole IR thing might be due to differences in CRTM coeffs:

1. when I try running JEDI pointing to GSI coeffs, JEDI die complaining of the following:

CRTM_AerosolCoeff_Load(FAILURE) : Error reading AerosolCoeff file /discover/nobackup/projects/gmao/dadev/rtodling/JEDI/x49/j49rt00/fvInput/gsi/etc/fix_ncep20221018/REL-2.4.0-jcsda/CRTM_Coeffs/Little_Endian/AerosolCoeff.bin
AerosolCoeff_ValidRelease(INFORMATION) : A AerosolCoeff data update is needed. AerosolCoeff release is 3. Valid release is 4.

2. when I try running GSI with the CRTM coeffs used in JEDI, GSI fails w/ the following error:

CRTM_Init(FAILURE) : Error loading IRwaterCoeff data from /discover/nobackup/projects/gmao/advda/SwellStaticFiles/je
di/crtm_coefficients/2.4.1/Nalli.IRwater.EmisCoeff.bin; Process ID: 4
crtm_interface*init_crtm: ERROR crtm_init error_status= 3

The funny thing is that months ago I had been able to run JEDI pointing to the GSI-CRTM coeffs ... obviously JEDI evolved and this is no longer possible - I know for a fact that one thing the changed in JEDI-CRTM are the Aerosol-related files.

Perhaps I can change JEDI to allow it to use the V4 aerosol files ... looking ...

[rtodling]

Thanks for Matt, I am not able to run GSI w/ exactly the same version of CRTM as that used in JEDI - pointing to the CRTM coeffs that JEDI uses. Here is figure of the increments for Tskin when using GSI as we normally have it in x0049 (top) and GSI running JEDI's version of CRTM (bottom):

minor differences over ice that would/could be of importance weren't the issues above way more prominent.

And here is the difference in the profile of temperature averaged in the way I showed above for other cases gsi(def) - gsi(jedi(crtm)).

Some differences in the upper atmosphere, but nothing to be too alarmed as the differences we are finding between GSI and JEDI.

[rtodling]

Just to give a general feel for the difference in handling antenna vs brightness temperature for some instruments / platforms in GSI, here is a profile fig (similar to calculations above) comparing two all obs exps w/ GSI when Ta is converted to Tb.

From this, it would seem the differences seen in the treatment of, say, ATMS-N20 in JEDI vs GSI would not all be explained by the Ta vs Tb story ... nonetheless, I will do two exps in GSI (ta vs tb) when using only ATMS-N20 for completion. it is possible that w/ multiple instruments at play, the signal and differences between Ta vs Tb get mixed and dimmed ...

[rgelaro]

Thanks, Ricardo. You may be right, that the signal for MW alone would likely be larger since those obs are affected most by the Ta/Tb diff.

[gmao-yzhu]

I think it could be explained by Ta and Tb difference. In Ricardo’s figure, the difference is mostly shown above 5mb, those are higher levels affected by ATMS channel 15. ATMS channel 15 is not bias corrected, and all of the other channels are bias corrected so attena correction is partially taken care of in those channels but not in channel 15. That’s why we see differences above 5mb. From: rtodling ***@***.***> Date: Monday, September 30, 2024 at 2:47 PM To: GEOS-ESM/swell ***@***.***> Cc: Zhu, Yanqiu (GSFC-6101) ***@***.***>, Assign ***@***.***> Subject: [EXTERNAL] [BULK] Re: [GEOS-ESM/swell] Something is amiss in handling IASI (perhaps all hyperspectral IR) (Issue #426) CAUTION: This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC. Just to give a general feel for the difference in handling antenna vs brightness temperature for some instruments / platforms in GSI, here is a profile fig (similar to calculations above) comparing two all obs exps w/ GSI when Ta is converted to Tb. tinc_allobs_tbXta.png (view on web)<https://github.com/user-attachments/assets/2b9acff1-9ebb-439d-9c4f-440f6c302d84> From this, it would seem the differences seen in the treatment of, say, ATMS-N20 in JEDI vs GSI would not all be explained by the Ta vs Tb story ... nonetheless, I will do two exps in GSI (ta vs tb) when using only ATMS-N20 for completion. — Reply to this email directly, view it on GitHub<#426 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ARUMEBX25MF2QJFRPVRBKLTZZGMDZAVCNFSM6AAAAABOG6HI4CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBTHEZDCMRSGM>. You are receiving this because you were assigned.Message ID: ***@***.***>