Add end of run restart functionality to MOM6 #133
Conversation
This commit adds end or run restart functionality to MOM6. The restart files are written when the stop_run alarm has been triggered.
Remove debug check that wrote to log.
This commit allows the user to set write_restart_at_endofrun in ufs.configure.*.IN allow CMEPS and MOM6 to create a restart at the end of the run even if it falls outside of the restart interval.
dpsarmie
changed the title
The call to read the write_restart_at_endofrun config option has been placed in the InitializeAdvertise subroutine to avoid the the check to be called at every timestep. Default for the EOR restart has been set to False.
|
@DeniseWorthen can you take a preliminary code review before I reach out NCAR side ? |
Fix issue where wrong alarm was being turned off.
Moving call to read config file for endofrun restart attribute to the InitailizeP0 subroutine.
|
@dpsarmie I made a try with latest UWM but replaed with your MOM6 branch. Somehow I am not getting what I expected. I used cpld_control_c48_intel as a template and changed restart_n from 12 to 18. The run length is 24hr and the IC is 20210306. With write_restart_at_endofrun=T it shall give us 20210400 and 20240406 ocean restart files. But I only see the frist one being written out. My run dir is on HERA: /scratch1/NCEPDEV/stmp2/Jiande.Wang/FV3_RT/rt_272814/cpld_control_c48_inte-T can you take a look ? is there anything I missed or is there a specific CMEPS branch that I shall use ? |
|
@jiandewang I believe the test you want to do is to leave restart_n=12 but make the run length 27. You should get restarts at hour 12,24 and 27. Without this 'end-of-run' setting, you would only get restarts at 12,24, even though you run all the way to 27. |
|
@DeniseWorthen in order to to extend to 27hr run, do I need to change stop_n to 27 (alone with model_configure nhours_fcst=27) ? |
|
@jiandewang stop_n should be set == to fhmax |
|
@just made a quick test, see /scratch1/NCEPDEV/stmp2/Jiande.Wang/FV3_RT/rt_272814/cpld_control_c48_inte-TDW still don't see final restart file |
|
@dpsarmie Could you please check Jiande's run and see if you can spot the issue? |
|
@jiandewang cpld_control_c48 uses parm/ufs.configure.s2s_esmf.IM but that file does not have the configuration option active. Go ahead and add write_restart_at_endofrun = .true. and rerun the test. That should (hopefully) solve the problem. |
|
@DeniseWorthen We can talk about whether or not we should add that option to the other ufs.configure files. Currently, only the HAFS and DATM ufs.configure files have the option in the configuration files. |
|
@dpsarmie but if you see /scratch1/NCEPDEV/stmp2/Jiande.Wang/FV3_RT/rt_272814/cpld_control_c48_inte-TDW/ufs.configure, I have that in line 124 |
Ok I see that. I'll try to run the c48 case and see if I can replicate the issue. |
|
@dpsarmie When we add this to UWM, we will want to add a configure variable to all the ufs.config files, but in the RT system, this could be false by default. The G-W will then be able to set it true if they need to. |
Sounds good. @jiandewang , I'm seeing that this is False in your mediator.log: Haven't found what could be causing the issue causing the mediator flag to be incorrectly set but I'll keep looking tomorrow. |
|
I think the issue might be that Jiande used "true" vs ".true." |
|
I have a "true" case queued up on Hera right now. I figured that the parser would handle either case correctly but I'll wait an see what this test shows. |
Ok. Denise is right though, I did a test run with "true" and it wasn't parsed correctly. I'll keep looking through your logs and see if there's any other issues. |
|
yes, just looking through some of the other logicals and it doesn't seem to matter. weird. |
|
just re-submitted my test case with .true. let's wait for couple of minutes to see what's going on |
|
using .true. give me what I am expecting. |
|
@jiandewang Please check that you also get mediator cpl.r files at the same times that you get MOM6 restarts. In the end, when also including the CICE changes, we need all three components to have the capability to write at restart_n and at the end. |
|
@DeniseWorthen yes we got that file for my test case, run length=30, restart_n=12 |
|
my second try: run length=60, restart_n=18, works as expected: /scratch1/NCEPDEV/stmp2/Jiande.Wang/FV3_RT/rt_272814/cpld_control_c48_inte-TDW3[186]ll RESTART/ufs* I am going to ask NACR to test on their side to make sure it won't break their system |
|
@dpsarmie MOM6 dev/emc just had one updating. Can you sync your branch ? I will reach out NCAR after you sync your branch. Thanks |
@jiandewang , I've updated the branch. Thanks again for testing and the help. |
|
now we got greenlight from NCAR. I will prepare a UWM PR for it to get merged to dev/emc |
|
combined with UWM (ufs-community/ufs-weather-model#2205) |
|
Testing for #2205 has completed successfully, please continue with merging this PR. |
This PR allows the user to create restart files at the end of a run in MOM using the write_restart_at_endofrun configuration option in CMEPS. This configuration option will control end of run restarts for MOM6, CICE, and CMEPS. This PR closes NOAA-EMC/CMEPS#118 and closes ufs-community/ufs-weather-model#2236. A similar PR was made in the CICE repository (CICE PR#77) that will completely resolve these issues.
The code was tested on Hera using the regression test datm_cdeps_control_gefs. This was tested using different combinations of restart setting to try and ensure expected functionality.
If the setting is not set or set to False, a restart file will not be created at the end of the run. Setting the option to true will create the file.
The end of file restarts for CMEPS, CICE, and MOM will all be controlled with this single configuration option.