Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changes to get C48 aerosol test ready #1829

Merged

Conversation

CoryMartin-NOAA
Copy link
Contributor

Description

This PR partially addresses #1821 in that it allows for a C48 simplified aerosol 3DVar test using the workflow to run in GDASApp CI testing. The workflow team will be able to adapt this test case for their CI testing with minimal effort.

This limits the C48 aerosol analysis to 6 PEs and updates the hash of GDASApp to a stable build from this morning.

Type of change

  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO

How has this been tested?

Built and tested on Hera

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • I have made corresponding changes to the documentation if necessary

@CoryMartin-NOAA
Copy link
Contributor Author

#1826 will depend on this as it will need the updated GDASApp hash

Copy link
Contributor

@RussTreadon-NOAA RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look good to me. Cloned CoryMartin-NOAA:feature/aero3dvar_test1 on Orion and will run test_gdasapp after building GDASApp.

Tentative approve.

@RussTreadon-NOAA
Copy link
Contributor

@CoryMartin-NOAA , test_gdasapp_aero_jjob_var_run failed on Orion with the following traceback

0: GeometryData: Global tree size = 13824, footprint = 884736
0: Exception:   Reason: An exception occurred inside ioda while opening a variable.
0:      name:   Location
0:      source_column:  0
0:      source_filename:        /work2/noaa/da/rtreadon/git/global-workflow/pr1829/sorc/gdas.cd/ioda/src/engines/ioda/src/ioda/Has_Variables.cpp
0:      source_function:        ioda::Variable ioda::detail::Has_Variables_Base::open(const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> &) const
0:      source_line:    108
0:
0: Exception: oops::Variational<FV3JEDI, UFO and IODA observations> terminating...
0: OOPS Ending   2023-08-30 17:24:40 (UTC+0000)

Obviously, test_gdasapp_aero_jjob_var_final failed because the output expected from the run job was not present.

@CoryMartin-NOAA
Copy link
Contributor Author

@RussTreadon-NOAA thanks, good catch, this is because I forgot to copy the new ob file that @andytangborn created to orion, I only put it on Hera. It should be updated now, all three (init, run, final) will need to be re-ran to confirm.

@RussTreadon-NOAA
Copy link
Contributor

Thank you @CoryMartin-NOAA for the quick fix. The run job now Passed but final Failed

(gdasapp) (gdasapp) (gdasapp) Orion-login-3:/work2/noaa/da/rtreadon/git/global-workflow/pr1829/sorc/gdas.cd/build$ ctest -R test_gdasapp_aero_jjob_var
Test project /work2/noaa/da/rtreadon/git/global-workflow/pr1829/sorc/gdas.cd/build
    Start 1629: test_gdasapp_aero_jjob_var_init
1/3 Test #1629: test_gdasapp_aero_jjob_var_init ....   Passed   43.32 sec
    Start 1630: test_gdasapp_aero_jjob_var_run
2/3 Test #1630: test_gdasapp_aero_jjob_var_run .....   Passed   74.13 sec
    Start 1631: test_gdasapp_aero_jjob_var_final
3/3 Test #1631: test_gdasapp_aero_jjob_var_final ...***Failed   10.15 sec

67% tests passed, 1 tests failed out of 3

Total Test time (real) = 128.13 sec

The following tests FAILED:
        1631 - test_gdasapp_aero_jjob_var_final (Failed)
Errors while running CTest
Output from these tests are in: /work2/noaa/da/rtreadon/git/global-workflow/pr1829/sorc/gdas.cd/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

Reran the final job with -VV and see a Permission denied in the log file

  File "/work2/noaa/da/rtreadon/git/global-workflow/pr1829/ush/python/pygfs/task/analysis.py", line 129, in add_fv3_increments
    with Dataset(inc_path, mode='r') as incfile, Dataset(bkg_path, mode='a') as rstfile:
  File "netCDF4/_netCDF4.pyx", line 2135, in netCDF4._netCDF4.Dataset.__init__
  File "netCDF4/_netCDF4.pyx", line 1752, in netCDF4._netCDF4._ensure_nc_success
PermissionError: [Errno 13] Permission denied: b'/work2/noaa/da/rtreadon/git/global-workflow/pr1829/sorc/gdas.cd/build/test/aero/global-workflow/testrun/ROTDIRS/gdas_test/gdas.20210323/12//model_data/atmos/restart/20210323.180000.fv_tracer.res.tile1.nc'

My restart directory is a soft link to /work2/noaa/da/cmartin/CI/GDASApp/data/lowres/gdas.20210323/12/model_data/atmos/restart

Orion-login-1:/work2/noaa/da/rtreadon/git/global-workflow/pr1829/sorc/gdas.cd$ ls -l /work2/noaa/da/rtreadon/git/global-workflow/pr1829/sorc/gdas.cd/build/test/aero/global-workflow/testrun/ROTDIRS/gdas_test/gdas.20210323/12/model_data/atmos/
total 4
lrwxrwxrwx 1 rtreadon da 87 Aug 30 17:46 restart -> /work2/noaa/da/cmartin/CI/GDASApp/data/lowres/gdas.20210323/12/model_data/atmos/restart

I can't update a file which you own.

@CoryMartin-NOAA
Copy link
Contributor Author

@RussTreadon-NOAA ah! this is why we have independent tests! So it needs to be a cp and not a symlink it seems.

@RussTreadon-NOAA
Copy link
Contributor

Tinkered with test/aero/global-workflow/jjob_var_init.sh as follows

-# Link model bacgkround on tiles
+# Copy model bacgkround on tiles
 dpath=gdas.$gPDY/$gcyc/model_data/atmos
 COM_ATMOS_RESTART_PREV_DIRNAME=$(dirname $COM_ATMOS_RESTART_PREV)
-mkdir -p $COM_ATMOS_RESTART_PREV_DIRNAME
-flist="restart"
+if [ -d $COM_ATMOS_RESTART_PREV_DIRNAME/restart ]; then
+    rm -rf $COM_ATMOS_RESTART_PREV_DIRNAME/restart
+fi
+mkdir -p $COM_ATMOS_RESTART_PREV_DIRNAME/restart
+flist="restart/*"
 for file in $flist; do
-   ln -fs $GDASAPP_TESTDATA/lowres/$dpath/$file $COM_ATMOS_RESTART_PREV_DIRNAME/
+   cp $GDASAPP_TESTDATA/lowres/$dpath/$file $COM_ATMOS_RESTART_PREV_DIRNAME/restart/
 done

With this change all three aero tests Passed

(gdasapp) (gdasapp) (gdasapp) Orion-login-3:/work2/noaa/da/rtreadon/git/global-workflow/pr1829/sorc/gdas.cd/build$ ctest -R test_gdasapp_aero_jjob
Test project /work2/noaa/da/rtreadon/git/global-workflow/pr1829/sorc/gdas.cd/build
    Start 1629: test_gdasapp_aero_jjob_var_init
1/3 Test #1629: test_gdasapp_aero_jjob_var_init ....   Passed   13.12 sec
    Start 1630: test_gdasapp_aero_jjob_var_run
2/3 Test #1630: test_gdasapp_aero_jjob_var_run .....   Passed   74.14 sec
    Start 1631: test_gdasapp_aero_jjob_var_final
3/3 Test #1631: test_gdasapp_aero_jjob_var_final ...   Passed   74.16 sec

100% tests passed, 0 tests failed out of 3

Total Test time (real) = 161.96 sec

@RussTreadon-NOAA
Copy link
Contributor

Pull in updated hash and rerun test_gdasapp_aero_jjob. All three tests pass.

Test project /work2/noaa/da/rtreadon/git/global-workflow/pr1829/sorc/gdas.cd/build
    Start 1629: test_gdasapp_aero_jjob_var_init
1/3 Test #1629: test_gdasapp_aero_jjob_var_init ....   Passed  117.79 sec
    Start 1630: test_gdasapp_aero_jjob_var_run
2/3 Test #1630: test_gdasapp_aero_jjob_var_run .....   Passed   74.14 sec
    Start 1631: test_gdasapp_aero_jjob_var_final
3/3 Test #1631: test_gdasapp_aero_jjob_var_final ...   Passed  106.15 sec

100% tests passed, 0 tests failed out of 3

Total Test time (real) = 298.84 sec

Copy link
Contributor

@aerorahul aerorahul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good.

@WalterKolczynski-NOAA WalterKolczynski-NOAA merged commit 573ecce into NOAA-EMC:develop Aug 31, 2023
3 checks passed
@CoryMartin-NOAA CoryMartin-NOAA deleted the feature/aero3dvar_test1 branch June 14, 2024 18:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants