Changes to get C48 aerosol test ready

[CoryMartin-NOAA]

Description

This PR partially addresses #1821 in that it allows for a C48 simplified aerosol 3DVar test using the workflow to run in GDASApp CI testing. The workflow team will be able to adapt this test case for their CI testing with minimal effort.

This limits the C48 aerosol analysis to 6 PEs and updates the hash of GDASApp to a stable build from this morning.

Type of change

• Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? NO

How has this been tested?

Built and tested on Hera

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes generate no new warnings
• New and existing tests pass with my changes
• I have made corresponding changes to the documentation if necessary

[CoryMartin-NOAA]

#1826 will depend on this as it will need the updated GDASApp hash

[RussTreadon-NOAA]

Changes look good to me. Cloned CoryMartin-NOAA:feature/aero3dvar_test1 on Orion and will run test_gdasapp after building GDASApp.

Tentative approve.

[RussTreadon-NOAA]

@CoryMartin-NOAA , test_gdasapp_aero_jjob_var_run failed on Orion with the following traceback

0: GeometryData: Global tree size = 13824, footprint = 884736
0: Exception:   Reason: An exception occurred inside ioda while opening a variable.
0:      name:   Location
0:      source_column:  0
0:      source_filename:        /work2/noaa/da/rtreadon/git/global-workflow/pr1829/sorc/gdas.cd/ioda/src/engines/ioda/src/ioda/Has_Variables.cpp
0:      source_function:        ioda::Variable ioda::detail::Has_Variables_Base::open(const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> &) const
0:      source_line:    108
0:
0: Exception: oops::Variational<FV3JEDI, UFO and IODA observations> terminating...
0: OOPS Ending   2023-08-30 17:24:40 (UTC+0000)

Obviously, test_gdasapp_aero_jjob_var_final failed because the output expected from the run job was not present.

[CoryMartin-NOAA]

@RussTreadon-NOAA thanks, good catch, this is because I forgot to copy the new ob file that @andytangborn created to orion, I only put it on Hera. It should be updated now, all three (init, run, final) will need to be re-ran to confirm.

[RussTreadon-NOAA]

Thank you @CoryMartin-NOAA for the quick fix. The run job now Passed but final Failed

(gdasapp) (gdasapp) (gdasapp) Orion-login-3:/work2/noaa/da/rtreadon/git/global-workflow/pr1829/sorc/gdas.cd/build$ ctest -R test_gdasapp_aero_jjob_var
Test project /work2/noaa/da/rtreadon/git/global-workflow/pr1829/sorc/gdas.cd/build
    Start 1629: test_gdasapp_aero_jjob_var_init
1/3 Test #1629: test_gdasapp_aero_jjob_var_init ....   Passed   43.32 sec
    Start 1630: test_gdasapp_aero_jjob_var_run
2/3 Test #1630: test_gdasapp_aero_jjob_var_run .....   Passed   74.13 sec
    Start 1631: test_gdasapp_aero_jjob_var_final
3/3 Test #1631: test_gdasapp_aero_jjob_var_final ...***Failed   10.15 sec

67% tests passed, 1 tests failed out of 3

Total Test time (real) = 128.13 sec

The following tests FAILED:
        1631 - test_gdasapp_aero_jjob_var_final (Failed)
Errors while running CTest
Output from these tests are in: /work2/noaa/da/rtreadon/git/global-workflow/pr1829/sorc/gdas.cd/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

Reran the final job with -VV and see a Permission denied in the log file

  File "/work2/noaa/da/rtreadon/git/global-workflow/pr1829/ush/python/pygfs/task/analysis.py", line 129, in add_fv3_increments
    with Dataset(inc_path, mode='r') as incfile, Dataset(bkg_path, mode='a') as rstfile:
  File "netCDF4/_netCDF4.pyx", line 2135, in netCDF4._netCDF4.Dataset.__init__
  File "netCDF4/_netCDF4.pyx", line 1752, in netCDF4._netCDF4._ensure_nc_success
PermissionError: [Errno 13] Permission denied: b'/work2/noaa/da/rtreadon/git/global-workflow/pr1829/sorc/gdas.cd/build/test/aero/global-workflow/testrun/ROTDIRS/gdas_test/gdas.20210323/12//model_data/atmos/restart/20210323.180000.fv_tracer.res.tile1.nc'

My restart directory is a soft link to /work2/noaa/da/cmartin/CI/GDASApp/data/lowres/gdas.20210323/12/model_data/atmos/restart

Orion-login-1:/work2/noaa/da/rtreadon/git/global-workflow/pr1829/sorc/gdas.cd$ ls -l /work2/noaa/da/rtreadon/git/global-workflow/pr1829/sorc/gdas.cd/build/test/aero/global-workflow/testrun/ROTDIRS/gdas_test/gdas.20210323/12/model_data/atmos/
total 4
lrwxrwxrwx 1 rtreadon da 87 Aug 30 17:46 restart -> /work2/noaa/da/cmartin/CI/GDASApp/data/lowres/gdas.20210323/12/model_data/atmos/restart

I can't update a file which you own.

[CoryMartin-NOAA]

@RussTreadon-NOAA ah! this is why we have independent tests! So it needs to be a cp and not a symlink it seems.

[RussTreadon-NOAA]

Tinkered with test/aero/global-workflow/jjob_var_init.sh as follows

-# Link model bacgkround on tiles
+# Copy model bacgkround on tiles
 dpath=gdas.$gPDY/$gcyc/model_data/atmos
 COM_ATMOS_RESTART_PREV_DIRNAME=$(dirname $COM_ATMOS_RESTART_PREV)
-mkdir -p $COM_ATMOS_RESTART_PREV_DIRNAME
-flist="restart"
+if [ -d $COM_ATMOS_RESTART_PREV_DIRNAME/restart ]; then
+    rm -rf $COM_ATMOS_RESTART_PREV_DIRNAME/restart
+fi
+mkdir -p $COM_ATMOS_RESTART_PREV_DIRNAME/restart
+flist="restart/*"
 for file in $flist; do
-   ln -fs $GDASAPP_TESTDATA/lowres/$dpath/$file $COM_ATMOS_RESTART_PREV_DIRNAME/
+   cp $GDASAPP_TESTDATA/lowres/$dpath/$file $COM_ATMOS_RESTART_PREV_DIRNAME/restart/
 done

With this change all three aero tests Passed

(gdasapp) (gdasapp) (gdasapp) Orion-login-3:/work2/noaa/da/rtreadon/git/global-workflow/pr1829/sorc/gdas.cd/build$ ctest -R test_gdasapp_aero_jjob
Test project /work2/noaa/da/rtreadon/git/global-workflow/pr1829/sorc/gdas.cd/build
    Start 1629: test_gdasapp_aero_jjob_var_init
1/3 Test #1629: test_gdasapp_aero_jjob_var_init ....   Passed   13.12 sec
    Start 1630: test_gdasapp_aero_jjob_var_run
2/3 Test #1630: test_gdasapp_aero_jjob_var_run .....   Passed   74.14 sec
    Start 1631: test_gdasapp_aero_jjob_var_final
3/3 Test #1631: test_gdasapp_aero_jjob_var_final ...   Passed   74.16 sec

100% tests passed, 0 tests failed out of 3

Total Test time (real) = 161.96 sec

[RussTreadon-NOAA]

Pull in updated hash and rerun test_gdasapp_aero_jjob. All three tests pass.

Test project /work2/noaa/da/rtreadon/git/global-workflow/pr1829/sorc/gdas.cd/build
    Start 1629: test_gdasapp_aero_jjob_var_init
1/3 Test #1629: test_gdasapp_aero_jjob_var_init ....   Passed  117.79 sec
    Start 1630: test_gdasapp_aero_jjob_var_run
2/3 Test #1630: test_gdasapp_aero_jjob_var_run .....   Passed   74.14 sec
    Start 1631: test_gdasapp_aero_jjob_var_final
3/3 Test #1631: test_gdasapp_aero_jjob_var_final ...   Passed  106.15 sec

100% tests passed, 0 tests failed out of 3

Total Test time (real) = 298.84 sec

[aerorahul]

looks good.