Enable clearsky TB / radiance output under all conditions, even if clear

[BenjaminTJohnson]

From @ibanos90:
The brightness temperature "assuming clear sky" (var_tb_clr in the ufo variables) is the clear-sky background brightness temperature without considering cloud-scattering (Tb_clear). In ufo/src/ufo/operators/crtm/ufo_crtm_passive_mod.F90 (in the develop branch), says that Tb_Clear requires CRTM_Atmosphere_IsFractional(cloud_coverage_flag) to be true, which happens when atm(jprofile)%Cloud_Fraction > MIN_COVERAGE_THRESHOLD (1e.-6). However, in situations where the atm(jprofile)%Cloud_Fraction is zero (< 1e.-6) then Tb_clear is zero (initial value).

RTSolution(ln,m)%Tb_Clear is initialized with zero values and if CRTM_Atmosphere_IsFractional(cloud_coverage_flag) is True, then RTSolution(ln,m)%Tb_Clear = RTSolution_Clear%Brightness_Temperature (https://github.com/JCSDA-internal/crtm/blob/release/crtm_jedi/libsrc/CRTM_K_Matrix_Module.f90#L1114).

The purpose of this issue is to discuss the requirements for clear-sky output from CRTM, and under what conditions it is to be used.

I've also attached the cloud_fraction document.
Cloud_Fraction.pdf

[gthompsnJCSDA]

@BenjaminTJohnson @chengdang I have some new results using the atm_k%Radiance and atm_k%R_clear variables in the Jacobian part of my UFO branch with visible reflectance and it shows that areas completely free of clouds are retaining the pre-initialized value of ONE instead of having a good value of final albedo_assuming_clear_sky. It would be wonderful if this issue could be addressed in time for Skylab_V8 to resolve.

Addendum: Within my YAML, I have hardwired Cloud_Fraction: 1.0 so I was thinking that would assume all columns were 100% cloudy.

[chengdang]

I have some new results using the atm_k%Radiance and atm_k%R_clear variables in the Jacobian part of my UFO branch with visible reflectance and it shows that areas completely free of clouds are retaining the pre-initialized value of ONE instead of having a good value of final albedo_assuming_clear_sky

Hello Greg, do you mean the albedo_assuming_clear_sky is one for clear sky cases, and in these cases albedo_assuming_clear_sky is competed with atm_k%Radiance and atm_k%R_clear? Thank you.

FYI: this may be a typo, only rts_k has variables Radiance and R_clear

[gthompsnJCSDA]

@chengdang Please have a look at the 10 lines below this comment line in my feature branch:
https://github.com/JCSDA-internal/ufo/blob/ba4d62e0545f3d1da89df9a0b8851ce89151fdd7/src/ufo/operators/crtm/ufo_crtm_passive_mod.F90#L240

That is how I am filling the hofxdiags for albedo_assuming_clear_sky and my plot of this variable in the Jacobian file shows an albedo of 1.0 only in places that are 0% cloud fraction I think. But I am not sure and also not sure how to debug but the plot looks sorta obvious.

[weiwilliam]

@gthompsnJCSDA I guess it is something related to the cloud_coverage_flag defined by CRTM_Atmosphere_Coverage in src/Atmosphere/CRTM_Atmosphere.f90,
If the column is water content free, it will still be identified as CLEAR.

CRTMv3/src/Atmosphere/CRTM_Atmosphere.f90

Lines 179 to 181 in 7ce0c0e

	cloudy_layer_mask = atm%Cloud(n)%Water_Content > WATER_CONTENT_THRESHOLD
	nc = COUNT(cloudy_layer_mask)
	IF ( nc == 0 ) CYCLE Cloud_Loop

So, R_Clear won't be filled by RTSolution_Clear%Radiance, because CRTM_Atmosphere_IsFractional(cloud_coverage_flag) is not true.

CRTMv3/src/CRTM_K_Matrix_Module.f90

Lines 1432 to 1436 in 7ce0c0e

	IF ( SpcCoeff_IsVisibleSensor( SC(SensorIndex) ) .OR. &
	SpcCoeff_IsUltravioletSensor( SC(SensorIndex) ) ) THEN
	IF ( CRTM_Atmosphere_IsFractional(cloud_coverage_flag) ) THEN
	RTSolution(ln,m)%R_Clear = RTSolution_Clear(nt)%Radiance
	RTSolution(ln,m)%Tb_Clear = RTSolution_Clear(nt)%Brightness_Temperature

[chengdang]

Addendum: Within my YAML, I have hardwired Cloud_Fraction: 1.0 so I was thinking that would assume all columns were 100% cloudy.

Adding discussion to this, following the comment by @weiwilliam, even if Cloud_Fraction is set as 1.0, if geoval files do not pass valid radius, temperature and etc to CRTM, there is no cloud-related calculation either.

[chengdang]

@gthompsnJCSDA, I guess in summary, would you like CRTM to output valid rts%R_clear and etc, even if the cloud fraction is 0? Just like Ben described in this issue?

[gthompsnJCSDA]

@gthompsnJCSDA, I guess in summary, would you like CRTM to output valid rts%R_clear and etc, even if the cloud fraction is 0? Just like Ben described in this issue?

I am pretty sure that's the point of creating the issue.

[chengdang]

@gthompsnJCSDA, I guess in summary, would you like CRTM to output valid rts%R_clear and etc, even if the cloud fraction is 0? Just like Ben described in this issue?

I am pretty sure that's the point of creating the issue.

Good. It's not clear with your initial description of atm_k.

[gthompsnJCSDA]

I have a solution that is purely a "band aid" not a very permanent solution. Using something that @emilyhcliu started, we can trick CRTM to give the clear and cloudy-sky result by using an additional UFO yaml option in the operator. Indented at the same level you would find Cloud_Fraction: 1.0 a user can also use Cloud_Seeding: true and it will cause UFO to pass very tiny values of water species mixing ratio to CRTM. That amount is trivially small and won't affect clear sky calculations in any sizable way. Doing this causes CRTM not to give a missing value of the clear sky. @liujake this will solve your issue within MPAS as well.

A graphic of the in-development visible albedo when this trick is used is shown below. Without that Cloud_Seeding option, there would be areas of pure zero water species that output albedo of 1.0.

[ibanos90]

Hi @gthompsnJCSDA, thanks for providing this solution. I will test it and let you know how it goes.

[ibanos90]

@gthompsnJCSDA @liujake @BenjaminTJohnson Results from an HofX run with GOES16-ABI observations for 00Z 15 Apr 2018 using Cloud_Seeding: true show that this solution helps to reduce the amount of Tb_clear with zero values compared to results from a run without this option. However, as you can see in the figures below, it does not resolve the issue entirely (note that we use cloud fraction diagnostic from MPAS model). The first two panels show the Tb_clear results (scatter plot map and histogram) without the Cloud_seeding and the second two panels show results with the Cloud_seeding parameter activated, at all observations locations.

[gthompsnJCSDA]

This is kinda a bummer. My test was complete filling in of the "holes" of data when I added it for visible albedo. I will attempt to re-try the IR channels. I would have expected it to work as well as it did for me.

[emilyhcliu]

This is kinda a bummer. My test was complete filling in of the "holes" of data when I added it for visible albedo. I will attempt to re-try the IR channels. I would have expected it to work as well as it did for me.

Same for me. When I set cloud_seeding to true, it fixed the zero Tb_clear problem.

[BenjaminTJohnson]

@gthompsnJCSDA @ibanos90 please let me know how your tests go -- doing this in CRTM has turned out to be a massive effort. I think that the best way is to simply run CRTM twice for the same profile, but removing clouds and aerosols in the "clear sky" run. I do want to point out that the clear sky radiances computed by CRTM are cleared of both aerosol and cloud influences, BUT only when there's a cloud. When there's no cloud, the aerosol effects are still computed. This results in some potential confusion when attempting to use this for IR sensors. The cloud fraction capability was developed primarily for MW sensors, where aerosols are not considered. So my recommendation is if you want truly clear sky scenes (no clouds, no aerosols) under all circumstances you need to run CRTM with the concentrations of those species set to zero.

[gthompsnJCSDA]

@BenjaminTJohnson I understand what you're saying. I think my "band aid" fix might be considered a real fix that means we do not have to take the added step of running profiles twice. The minute amounts of each hydrometeor species along with the 0.1% cloud fraction assumption should be as good as running a second time with true zero values. We could even cross check that pretty easily I think. Meanwhile, we could also do very similar with aerosols by doing giving fake epsilon values for any number of aerosols just like with hydrometeors. I await word that @ibanos90 can confirm my solution as well, but I have a high confidence this is it.

BTW, this is also needed to test with a microwave instrument but I suspect it will do the trick nicely there. All-sky MW should be made better with this I believe - so long as we ensure the QC steps are set appropriately because some clear sky pixels would drastically change with rain/graupel hydrometeors given non-zero. But no one has yet (to my knowledge) set Cloud_Seeding: true in those yaml files.

[emilyhcliu]

@gthompsnJCSDA @ibanos90 please let me know how your tests go -- doing this in CRTM has turned out to be a massive effort. I think that the best way is to simply run CRTM twice for the same profile, but removing clouds and aerosols in the "clear sky" run. I do want to point out that the clear sky radiances computed by CRTM are cleared of both aerosol and cloud influences, BUT only when there's a cloud. When there's no cloud, the aerosol effects are still computed. This results in some potential confusion when attempting to use this for IR sensors. The cloud fraction capability was developed primarily for MW sensors, where aerosols are not considered. So my recommendation is if you want truly clear sky scenes (no clouds, no aerosols) under all circumstances you need to run CRTM with the concentrations of those species set to zero.

@BenjaminTJohnson What you suggested here is what we are doing in GSI. For all-sky assimilation, we call K-matrix, and we call the forward model for Tb_clear calculation. This avoids zero Tb_clear from the K-matrix call.