CICE writing extra restarts

[anton-seaice]

TL;DR: The CICE cmeps driver is not using the restart settings from nuopc.runconfig and is using those from ice_in

Using 1deg_jra55do_iaf I set:

 restart_n = 5
 restart_option = nyears
 stop_n = 5
 stop_option = nyears

in nuopc.runconfig

I then ran access-om3 (access-om3-d6813d6b9e1df560ac3f6ba6a605daab9cfd9569_main-5pjh7z2) using payu 1.1.3. The model completed and ran succesfully.

The archive/output000 folder incorrectly contained:

archive/output000/access-om3.cice.r.1959-01-01-00000.nc  archive/output000/access-om3.cice.r.1960-01-01-00000.nc  archive/output000/access-om3.cice.r.1961-01-01-00000.nc  archive/output000/access-om3.cice.r.1962-01-01-00000.nc

The correct restart was saved here:

archive/restart000/access-om3.cice.r.1963-01-01-00000.nc

The PBS error log also has this error from:

./archive_cice_restarts.sh: line 4: [: too many arguments

This is because CICE is writing annual restarts due to these settings (from ice.log):

dumpfreq = y x x x x (set in ice_in)
dumpfreq_n = 1 1 1 1 1 (the default)

I think the desired behaviour in the cmeps driver for CICE ignores/replaces and values from ice_in and sets them from the values in nuopc.runconfig ?

[ofa001]

Hi @anton-seaice I don't understand enough about the cmeps driver to be of help here Anton, I guess I am used to everything being under control of what's in ice_in hence CICE settings, but I guess across a coupled system we want to try and control it the same way the cmeps system has set it up and have similar approach to how MOM6 is controlled. I doubt we will be able to control the UM through cmeps though Martin Dix/Kieran Ricardo might know more.

[anton-seaice]

Thanks Siobhan. I guess the advantage here is we should only need to set the restart interval in one place (nuopc.runconfig) and then it is propagated consistently to MOM/CICE/WW + the mediator without needing to set the same information in multiple places.

[ofa001]

Yes if we can set it once for the CICE6-MOM6-WW3 set up that would be an advantage, so that it runs the same in all 3 models.

The coupling with the UM cmeps side will be a more complex set up that I am not fully across, just helping out with the fluxes, coupling fields and looking at the output side.

[MartinDix]

Our intention for CM3 is that settings in the rose/cylc suite set CMEPS and model control namelists consistently. We did this with OASIS in CM2 with driver scripts that modified the namelists.

[anton-seaice]

In ice_in, I suggest we change to dumpfreq = x to turn off annual restarts, as dump_last=.true. will ensure it writes a restart at the end of the run.

Its hard to map the restart_option and restart_n from nuopc.runconfig directly to CICE setup, because CICE allows up to 5 restart frequencies, and cmeps only has one. There is also write_restart_at_endofrun in cmeps, which doesn't seem to be supported in CICE or MOM (and end_restart)

We can add a note to nuopc.runconfig that dumpfreq in CICE needs to be set in ice_in if you want to set restart_n at a frequency other than the run length? (Alternatively, we could throw an error if the restart frequencies are different? I don't want to just overwrite the settings in ice_in with the value from nuopc.runconfig because this might be unclear to the end-user)

[ofa001]

Hi @anton-seaice This sounds like a sensible compromise, I guess when the model is doing longer runs under payu or rose/cylc control the values will also be controlled with settings from there to be across all the component models together, and in the test phase, users may be setting different options, but need to know if they are set centrally in nuopc.runconfig.

[anton-seaice]

Thanks @ofa001 - my assumption is most of the time, users won't want to write "extra" restart files, i.e. other than at the end of a run? If during longer runs writing extra restarts without restarting the model is common, then maybe its not the best compromise?

[ofa001]

Hi @anton-seaice, during test runs we tend to save more restarts, it also is useful if we want to go back and get additional high frequency data. Sub daily say, for downscaling in the atmosphere, hourly for ice stats in ice and ocean. However its often the other way we often want to clean up runs ( like we are doing on ik11) and restarts are the first thing that we clean up in long production runs, drop back to 10, 20 year intervals.

So I guess choose I sensible setting but document on how to change it relatively easily.

[anton-seaice]

It looks like CESM might put in a fix too , which means we can get all the settings through the CICE CMEPS driver. ESCOMP/CICE#29

[anton-seaice]

UFS have put a fix in:

CICE-Consortium/CICE#969

In future .write_restart_at_endofrun. will force a restart at the end of the RUN for CICE. Not sure if mom support is planned also.