Failure to restart reproducibly / time manager may be off one step - MPAS-O and MPAS-SI

[gdicker1]

Runs involving the MPAS-O and MPAS-SI components in EarthWorks that begin from restart files of a previous run do not produce the same results.

For example, an exact restart test of the CHAOS200dev compset fails with differences in 245 of 334 fields. In exact restart tests the model runs forward for 11 days (the "base" run), creates restart files at day 6, another run uses the restart files to run for days 6 through 11 (the "restart" run), and then outputs on day 11 between the base and restart runs are compared.

See the issues in the EarthWorksOrg repositories for other discussion:

• Failure to restart reproducibly / time manager may be off one step mpas-ocean#12
• Failure to restart reproducibly / time manager may be off one step mpas-seaice#15

Quoted material from mpas-ocean Issue 12 by @dazlich:

... I believe the time manager issues are likely responsible for the failure of the runs with an mpas-ocean component to restart reproducibly.

The time manager issues manifest themselves in the diagnostic output:

1. many types of hist files have an extra file with a time stamp of the month prior to the run start. This file generally has just one time step contributing to its time mean.
2. The hist files tagged with the current month are one time step shy in their contribution to the last time mean in the file. For instance, a January monthly mean will have only 743 hours contributing to it, and the January 31 daily mean will only have 23 hours contributing to it.
   Since restarted runs are close but do not quite reproduce a longer run I suspect the time manager issues are responsible here as well.

...