Does GEOSldas still require HDF4?

[mathomp4]

A query for @gmao-rreichle. Recently, I've been doing some work to try and get ifx support into GEOS, Baselibs, and the GEOS CI since by the end of this year Intel will be killing ifort as a Fortran compiler.

The main issue at the moment for LDAS is that the FORTRAN 77 interfaces to HDF4 are deprecated¹ and even if you force HDF4 to build them (as I do in Baselibs) even that does not work with ifx. So for now in my testing, if you build Baselibs with ifx, HDF4 is not built.

Now, I believe eventually the HDF Group plans to add "back" Fortran support with modern Fortran (see HDFGroup/hdf4#408 by @derobins), so might just be a "temporary" issue.

But perhaps the HDF code in the LDAS (namely it seems to be an AMSR-E obs reader?) isn't needed but just kept for backwards-looking work? If so, then maybe if there is a point in time where we have to move to ifx but HDF4 4.4.0 is not out yet, we can, if nothing else, have CMake check for HDF4 support and if not found, #ifdef out the code?

Footnotes

1. Note: They might even be removed in HDF4 4.3.0. I haven't actually tried updating and testing that yet. So this issue might be an "all-compiler" issue rather than just "ifx". ↩

[gmao-rreichle]

@mathomp4, thanks, as always, for keeping on top of these things! Much appreciated. We are not currently using AMSR-E obs, so there is no immediate need for HDF4 compatibility as far as I can tell. The most recent Version 3 of the AMSR-E obs are in *.he5 format, which I assume is version 5 of hdf. (The hdf 5 file extension used in SMAP is *.h5.) If we ever use AMSR-E obs in the future, we'll adapt the reader to use the *.he5 files, so I think we can forget about HDF4 altogether.

[mathomp4]

@gmao-rreichle Ahh. Good to know. I might look at adding in some CMake where "if ifx, don't compile certain code".

Note: We aren't quite ready for ifx yet. I think GEOSgcm v11 is unhappy with it, but GEOSgcm v12 does like it, so we are close. We also have MAPL internal tests failing with ifx as well, so a ways off...but hopefully not too long. oneAPI 2025 will be ifort-free.

[gmao-rreichle]

PS: I suppose we'll need a PR that disables the (hdf4) AMSR-E reader? Since the hdf4 reader will be useful if/when we convert it to h5, we should just comment it out and add a note, rather than delete the reader altogether.

[mathomp4]

@gmao-rreichle I talked with @tclune and I might do something simple in CMake like:

option(ENABLE_HDF4_SUPPORT "Enables build of HDF4 code" OFF)

where we default to OFF and then if that is enabled by the user, set an #ifdef and try to compile the code.

I'll try and test something like that to see. Once I get something to build, I might ask @biljanaorescanin to do a quick test to make sure all runs.

[gmao-rreichle]

Thanks, @mathomp4. But once we use the newer compiler, wouldn't the build necessarily fail when the user turns OFF to ON? In other words, the user option would only work for as long as we're using the current compilers, right? If that's correct, then we would have to remember to remove the switch again once we upgrade to the new compiler. It may be simpler to just disable hdf4 support for good now by commenting out the offending code with an #if 0 statement.

[mathomp4]

Thanks, @mathomp4. But once we use the newer compiler, wouldn't the build necessarily fail when the user turns OFF to ON? In other words, the user option would only work for as long as we're using the current compilers, right? If that's correct, then we would have to remember to remove the switch again once we upgrade to the new compiler. It may be simpler to just disable hdf4 support for good now by commenting out the offending code with an #if 0 statement.

Hmm. Yeah, you are probably right. Simpler at least :) I'll give that a try with ifx ...

[gmao-rreichle]

@mathomp4: Thanks for putting together this PR. Unfortunately, we have a problem. I hadn't realized that hdf4 is also used to read MODIS snow cover fraction (SCF) obs, which was added in the past year and is needed for planned reanalysis products. A quick check suggests that MODIS data are still served in hdf format. Unless I missed something, we would have to (i) convert the MODIS files into nc4 or h5 and (ii) adapt the GEOSldas readers. Maybe the DAAC has tools that convert the data prior to downloading. Unless there's been progress in adapting the hdf interfaces to modern Fortran, I'm afraid we somehow have to find the time to implement the workaround.

cc: @lcandre2 @biljanaorescanin

[mathomp4]

@gmao-rreichle Ahhh. Okay. That is...unfortunate. Hmm.

If they don't mind the summons... @byrnHDF What is the current recommendation for accessing HDF4 data via Fortran? Do we need to use iso_c_binding to write our own interfaces into the C layer?

Or perhaps would it be better to convert to HDF5 in someway? Searching around, the HDF-EOS folks seem to have a page about this where they seem to recommend converting to netCDF-4:

For users who want to convert NASA HDF4 and HDF-EOS2 files to HDF5/netCDF-4 or even netCDF-3 files, we strongly suggest to users to first check the HDF4 to CF conversion toolkit since HDF4 to CF conversion toolkit will convert NASA HDF4/HDF-EOS2 files to HDF5/netCDF-4 and netCDF-3 files by following the CF conventions

I see references online to people using h4tonccf_nc4 for AMSU-A data:

https://docs.dart.ucar.edu/en/latest/observations/obs_converters/AIRS/convert_amsu_L1.html#advanced-microwave-sounding-unit-amsu-a-l1b-brightness-temperatures

It would be nice if the DAAC could do this for us on download...

[lcandre2]

@mathomp4, right now, I download MOD10C1, C61 & MYD10C1, C61 to a project directory. It could be worth a conversation about adding these collections to CSS and including the conversion with that transition.

[mathomp4]

@lcandre2 Do you have a sample HDF file I can fiddle with? I want to see if we can just grab that converter and run it. I'm not sure if a Centos exec will work on SLES15 but I guess we can see. (If not, fun with Docker I suppose!)

[lcandre2]

@mathomp4, @gmao-rreichle and I discussed. I am going to see if the DAAC can convert to nc4 with the appropriate info during download and first -- this would be the easiest way! I'll report back.

[lcandre2]

@mathomp4, @gmao-rreichle It looks like only GeoTIFF reformatting is supported during download for the MODIS CMGs. I did post a comment on the Earthdata forum to confirm that I didn't miss anything. But Matt, if you have time, feel free to test the HDF to nc4 conversion on any of the files here: /discover/nobackup/projects/S2SHMA/MODIS/MOD10C1_V61

[mathomp4]

@lcandre2 Well, I grabbed the latest h4tonccf_nc4, copied one of your files, ran it and it made something!

cp /discover/nobackup/projects/S2SHMA/MODIS/MOD10C1_V61/2024/MOD10C1.A2024040.061.hdf .
./h4tonccf_nc4 MOD10C1.A2024040.061.hdf MOD10C1.A2024040.061.nc4

If you or @gmao-rreichle want to look at it, it's in:

/discover/nobackup/mathomp4/MODISHDF4

Plotting with Panoply, it's like the fields are not identical after conversion. They look nigh identical, but it's like there is a tiny shift?