Fb ufs regtests

[aliabdolali]

Pull Request Summary

• Adding ww3_grib build and run within regtests/bin/run_test.
• Updating the required libraries on Hera and Orion using hpc-stack for ww3_grib in regtests/bin/matrix_ncep.
• Adding regtests/bin/test.comp to compare two runs from the same code version for b4b reproducibility check.
• Reintegrate the Fix for arctic grib files Fix for arctic grib files  #346 from https://github.com/NOAA-EMC/WW3/tree/production/GFS.v16
• Add the following regtests for ufs application.
• Separate the MPI and OMP/OMPH tests in matrix_divider_p.sh

1. global model with wind, ice, and current, taken from ufs setup

• adding the -C ESMF test for the 1 deg global model
• adding the restart reproducibility check
• adding the MPI task reproducibility check
• adding the thread reproducibility check
• adding the ww3_grib test with NCO NCEP2 swtich

2. Adding GFSv16 setup with wind (hourly), ice (daily), and current (3hr interval)

• adding the -C ESMF test for the GFSv16
• adding the restart reproducibility check
• adding the ww3_grib test with NCO NCEP2 swtich

For regtests/bin/matrix_ncep, the required modules for esmf and grib outputs are added (hpc-stack).
The regtests/bin/matrix.base is updated and the check statement for restart b4b reproducibility is added.
The regtests/bin/matrix_divider_p.sh is updated and the large ufs tests which requires a larger number of cpus are separated.

issues

#317
#214
#323
#322
#181
closed #345
closed #276
closed #366
closed #367

Description

This development is for large setups specifically for UFS application (esmf) and GFSv16.

• Reviewers: @JessicaMeixner-NOAA

Testing

two regression tests are added
ww3_ufs1.1
ww3_ufs.1.2 (GFSv16)
There should be no changes in the regression test except the known non-identical cases and ww3_tp2.17 due to the changes in the inputs.

• Which compiler / HPC you used to run the regression tests in the PR?
  Testes on Hera with hpc-stack libraries

An issue related to GFSv16 setup is added (see #417)

[JessicaMeixner-NOAA]

@aliabdolali two questions:

1. is there new output with the matrix output since changes have been made?
2. Do you have a location of the ufs1.2 with the operations branch output I could see to compare?
   Thanks for reducing the resources significantly.

[aliabdolali]

@aliabdolali two questions:

1. is there new output with the matrix output since changes have been made?
2. Do you have a location of the ufs1.2 with the operations branch output I could see to compare?
   Thanks for reducing the resources significantly.

I haven't run the whole test again.
I deleted the production one.
I checked a commit before ukmet-coupling and it was still problematic,
So I checked a commit from August
commit ca406df (HEAD)
Author: Jessica Meixner [email protected]
Date: Mon Aug 3 23:09:14 2020 -0400
and the results are reasonable.
/scratch2/COASTAL/coastal/save/Ali.Abdolali/Source2021/May30GFSv16/regtests/ww3_ufs1.2/work_a

[aliabdolali]

@JessicaMeixner-NOAA
I addressed the comments you made and I ran the full matrix again, with expected changes due to this development:

ww3_tp2.17
ww3_tp2.6

and known non-b4b tests:

mww3_test_03
mww3_test_07
ww3_tp2.10
ww3_tp2.15

matrixDiff.txt
matrixCompSummary.txt
matrixCompFull.txt

I think this is ready to be merged and if you think we should wait for gfsv16 related bug, I can propose to exculde ww3_ufs1.2 (gfsv16) out and merge this one. In this way, we integrate the fixes and developments and leave the problematic one remained to be added to the bug fix for #417

[JessicaMeixner-NOAA]

@aliabdolali I will review again today.

[JessicaMeixner-NOAA]

In the PR description, I could not tell what would make tp2.6 and tp2.17 change? Which updates cause those changes exactly?

There is the issue with the new ufs 1.1 and 1.2 test which has been document in issue #417, and shown to be not a cause of this addition but an existing bug and therefore I think this can be merged. One comment on the wind input for ufs 1.2 is that the ice fields are blocking the u,v values. Therefore if you use a different ice input or if it varies over time and does not match the wind input itself, someone might have issues. As long as that is known and thought to be acceptable, we can move forward with this PR.

[aliabdolali]

@JessicaMeixner-NOAA

• For ww3_tp2.17, I changed the coordinates of point outputs. all of them were previously on the boundary nodes, now they are within the domain. I'll add a note in the PR description.

• For the ww3_tp2.6 in matrixCompFull.txt:
  ww3_tp2.6/./work_pdlib (0 files differ)
  so it is ok.

• I'll add a note of the ufs1.2 issue (multi grids is not working as expected #417)

[JessicaMeixner-NOAA]

this is now 15mext

Also, I made this file: /scratch2/NCEPDEV/climate/Jessica.Meixner/forinputww3ufstest/gfs.20210401/00/atmos/wind.2021040100.nc which is from gfs output and has hourly wind for 2021040100, could we use this instead of the wind that is blocked out by ice?

[JessicaMeixner-NOAA]

@aliabdolali thanks for updating the wind. The new ufs regression tests run except for the issue which is documented in #417. The info file for both of those tests are now out of date with the changes that have occurred since originally added. After those are updated and any last regtests are run, this should be ready to be merged.

[aliabdolali]

I ran the full matrix again, with expected changes in the inputs for:
ww3_tp2.17
and known non-b4b tests:

mww3_test_03
mww3_test_07
ww3_tp2.10
ww3_tp2.15

matrixCompFull.txt
matrixCompSummary.txt
matrixDiff.txt