Use ACCESS-OM2 grid, topography and initial conditions

[dougiesquire]

This PR moves the configuration to ACCESS-OM2 1 deg grid, topography, and initial conditions - see COSIMA/access-om3#36. A Python script is also included for generating ESMF mesh files - this should possibly be moved elsewhere at some point?

A few things to note. If everyone's okay with it, I'll leave these to be addressed in subsequent PRs. I'll open issues for them once this PR is merged.

• The MOM temperature and salt initial conditions used by ACCESS-OM2 are conservative temperature and practical salinity. Here, we use these initial conditions as is, but use the MOM6 "WRIGHT" EOS, so treat them as though the temperatures are actually potential temperatures - see MOM6 equation of state COSIMA/access-om3#20 (comment).

• The temperature and salt initial conditions are on the same grid as the model grid, which means we should be able to run with TEMP_SALT_INIT_VERTICAL_REMAP_ONLY = True in MOM_input. However, I couldn't get this to work. We should revisit why this is. This works using an access-om3 executable, rather than the cime-built one I was playing with early in this PR. Possibly due to FMS version.

• I'm currently generating an ESMF mesh file for the MOM grid and providing that. This is problematic as FMS2 doesn't support meshes (we're currently using FMS1). The MOM6 NUOPC cap does provide a option to set use_mommesh = .false. which then generates an ESMF Grid internally from the MOM grid. However, this wasn't trivial to get working in this configuration. It is being used in other configurations, so it shouldn't be hard to sort out.

• I've unset rof2ocn_ice_rmapname and rof2ocn_liq_rmapname in nuopc.runconfig. This should mean these mappings are generated at run-time, but for efficiency we should probably pre-compute these (and other mappings).

• The ice initial conditions are generated from a 3hr run of the 1deg_jra55_ryf configuration. This probably doesn't require a follow-up issue, but noting anyway. See CICE6 initial condition COSIMA/access-om3#50.

• I've removed the step of remapping atm/rof forcing to an intermediate grid by setting the data model grids equal to the ocn/ice grid. This means the spatial remapping is done in CDEPS prior to temporal interpolation, which should be more efficient that performing the spatial remapping with CMEPS after time interpolation. However, it's not obviously possible atm to pass precomputed remapping weights to CDEPS like you can for CMEPS remapping. We should consider adding this as it will presumably make a big difference for the high resolution configurations. We should also confirm that CMEPS recognises when the source and destination grids are the same and doesn't try to remap in this case.

• I haven't yet made any effort to get the parameter settings the same between the ACCESS-OM2 and OM3 configurations.

[dougiesquire]

Apologies, the commit history is a bit of a mess

[aekiss]

Awesome, thanks @dougiesquire I'll look at this next week

[ezhilsabareesh8]

Thanks @dougiesquire. Looks good to me.

[micaeljtoliveira]

If it's okay with everyone, I would to wait a bit before merging this (and also #6). The reason is that I would like to tag a version of then main branch that corresponds to the "unchanged" CESM configuration as it was generated by @dougiesquire . The idea would be to also tag a version of the model that exactly reproduces the results obtained with the corresponding CESM executable. This would provide a clear starting point for all future developments and changes.

[dougiesquire]

Note, I've added some info to the wiki on how the driver initialisation parameters interact with the component initialisation parameters: https://github.com/COSIMA/access-om3/wiki/NUOPC-driver#component-initialisation

[aekiss]

Great, thanks @dougiesquire

[dougiesquire]

@aekiss, this is now failing on "continue" runs, so probably best to hold off reviewing until I've worked out why (I'm not exactly sure when this problem started, as most of my testing has just been to run a single "startup" run).

[dougiesquire]

But it's probably clearer to remove runtype = "initial" from ice_in

Just noting that I think this is actually needed. It is important to let the NUOPC cap set this parameter based on the NUOPC start_type - see https://github.com/COSIMA/access-om3/wiki/NUOPC-driver#cice6.

[dougiesquire]

MOM is crashing on day 22 of the second monthly run. This crash still occurs if I change the atm and rof model grids back to the original (192x94 and 360x180) grids, but it does not occur if I use ice initial conditions ice_ic = "default" rather than the restart I created from ACCESS-OM2. That said, I don't think there's anything wrong with these initial conditions as it makes it 21 days before crashing.

The MOM error is:

WARNING from PE    47: Extreme surface sfc_state detected: i= 341 j= 280 lon=  67.145 lat=  70.619 x=  60.500 y=  80.743 D= 1.1806E+01 SSH= 8.8014E+00 SST=-2.1004E+00 SSS= 3.8733E+01 U-= 2.3777E-02 U+= 0.0000E+00 V-= 3.8195E-02 V+= 0.0000E+00

FATAL from PE     0: There were a total of         1 locations detected with extreme surface values!

If I'm interpreting this correctly, this suggests a SSH of nearly 9m. The location of the issue is in the Kara Sea (TIL about the Kara Sea). See the magenta cross:

@aekiss the MOM baroclinic timestep is already quite a bit shorter than it is in ACCESS-OM2 1deg_jra55_ryf (1800 s vs 5400 s). I presume we don't want to reduce this further (ideally would we increase it?). Any thoughts on the best way forward?

[dougiesquire]

The issue still occurs (though 1 day later) after adding salinity restoring as in ACCESS-OM2.

[aekiss]

Hm, interesting. Is there anything unusual in the sea ice or wind at that location and time?

[aekiss]

Agreed that we'd ultimately like a longer baroclinic timestep, but does a shorter timestep fix the issue?

[aekiss]

We used Rayleigh damping in a few select locations in OM2 (see sec 3.2.7 here). At 0.1° (but not 1°) we needed damping in the nearby Kara Strait. But that's really an engineering fix, best avoided if we can.

[aekiss]

Is there anything odd in the topography in that region?

[aekiss]

I don't think we need to worry about fixing the crash in the Kara Sea before we merge this, as crashes like this will need ongoing tweaks to resolve, and we won't be running with this forcing anyway.

I did a test merge of this branch with Support-JRA55-do
here: https://github.com/COSIMA/MOM6-CICE6/tree/om3-test

This ran much longer (until 13 Oct in the first year), then failed at the head of the Persian Gulf with

WARNING from PE    31: Extreme surface sfc_state detected: i= 329 j= 194 lon=  48.500 lat=  26.524 x=  48.500 y=  26.524 D= 1.1806E+01 SSH= 1.0595E+01 SST= 2.5510E+01 SSS= 4.5002E+01 U-= 0.0000E+00 U+=-4.1033E-02 V-= 0.0000E+00 V+= 1.8850E-02

There's nothing unusual in the winds at that location/time. I'll investigate whether changing the barotropic split (DTBT) helps.

[dougiesquire]

Apologies, I was working on other things for most of last week. I'm happy to merge if whenever others are.

[aekiss]

I think @micaeljtoliveira still wants to tag a CESM branch first

[aekiss]

I've made a separate issue to record attempts to resolve the crashing issue COSIMA/access-om3#55

[dougiesquire]

There have been recent updates to the CICE NUOPC cap that add an additional check that the mask in the supplied mesh matches an internally generated mesh. After rebasing onto 1deg_jra55do_ryf, the configuration in this PR currently fails that check. I've been trying to explore why this is and I'll include details below for the purpose of having things documented in the open.

[micaeljtoliveira]

Can I force push to your branch once #6 is merged?

Go ahead

@dougiesquire Done. Would you mind having a look and check I didn't mess up something?

[dougiesquire]

Ran this for 1 day without issue. A couple of questions:

• I know we discussed this in the recent TWG meeting, but there's a huge amount of input data that isn't actually being used by this configuration. This muddies the manifests and makes it difficult to work out which inputs are required and which aren't. Do you mind if I push a change to config.yaml to make the inputs a little more explicit?
  ADDED: Can you explain to me how the input data will be structured in /g/data/ik11/inputs/access-om3. Different configurations will have different inputs. Do we need 1deg etc subdirectories in each version directory?
  I've restructured the inputs directory and removed unused inputs from the config. I've also opened this issue in case we want to discuss the structure of the inputs directories further.

• I see you've copied the input files I'm using from ACCESS-OM2 into /g/data/ik11/inputs/access-om3/0.x.0, rather than symlinking their original location in /g/data/ik11/inputs/access-om2. Can I change these to symlinks to keep track of their provenance?

• I originally reorganised MOM_input for convenience in finding/understanding parameters, but is the added convenience outweighed by added frustration when trying to incorporate upstream changes/additions?

[micaeljtoliveira]

I see you've copied the input files I'm using from ACCESS-OM2 into /g/data/ik11/inputs/access-om3/0.x.0, rather than symlinking their original location in /g/data/ik11/inputs/access-om2. Can I change these to symlinks to keep track of their provenance?

I discussed this with @aekiss last week and we agreed it would be better to fully separate the two sets of inputs (OM2 will be around for a long time, but probably not as long as OM3). Instead, we should add a README explaining the provenance of the files. We should probably do the same for the cime inputs by the way. But happy to discuss this further.

I originally reorganised MOM_input for convenience in finding/understanding parameters, but is the added convenience outweighed by added frustration when trying to incorporate upstream changes/additions?

I think it should be okay. The number of changes/additions from upstream that can/should be added to our configurations for any given CESM update is probably going to be small. so it can be done manually (ie. with a copy-paste instead of applying a patch). Probably it's actually better to do it manually, as someone should review them carefully.

[dougiesquire]

We should probably do the same for the cime inputs by the way. But happy to discuss this further.

Just noting that this PR actually does not use the cime inputs at all, but they are probably still needed for other configs so I agree we (I) should make note of their provenance

[micaeljtoliveira]

Just noting that this PR actually does not use the cime inputs at all

Great! I was actually wondering about that, but forgot to ask.

[aekiss]

The MOM_input changes are hard to follow as the order has also been rearranged. For the record, here are all the semantic differences as reported by nmltab:

* &null  adjust_net_srestore_to_zero                                                MOM_input_1deg_jra55do_ryf
* &null  adjust_net_srestore_to_zero         True                                   MOM_input_om2_grid_iss36
* &null  ale_coordinate_config               'FILE:ocean_vgrid.nc,dz'               MOM_input_1deg_jra55do_ryf
* &null  ale_coordinate_config               'FILE:ocean_vgrid.nc,interfaces=zeta'  MOM_input_om2_grid_iss36
* &null  diag_coord_def_z                    'WOA09'                                MOM_input_1deg_jra55do_ryf
* &null  diag_coord_def_z                    'FILE:ocean_vgrid.nc,interfaces=zeta'  MOM_input_om2_grid_iss36
* &null  eqn_of_state                                                               MOM_input_1deg_jra55do_ryf
* &null  eqn_of_state                        'WRIGHT'                               MOM_input_om2_grid_iss36
* &null  fluxconst                           0.5                                    MOM_input_1deg_jra55do_ryf
* &null  fluxconst                           0.11                                   MOM_input_om2_grid_iss36
* &null  grid_file                           'ocean_hgrid_230424.nc'                MOM_input_1deg_jra55do_ryf
* &null  grid_file                           'ocean_hgrid.nc'                       MOM_input_om2_grid_iss36
* &null  mask_srestore_under_ice                                                    MOM_input_1deg_jra55do_ryf
* &null  mask_srestore_under_ice             False                                  MOM_input_om2_grid_iss36
* &null  max_delta_srestore                                                         MOM_input_1deg_jra55do_ryf
* &null  max_delta_srestore                  0.5                                    MOM_input_om2_grid_iss36
* &null  niglobal                            320                                    MOM_input_1deg_jra55do_ryf
* &null  niglobal                            360                                    MOM_input_om2_grid_iss36
* &null  njglobal                            384                                    MOM_input_1deg_jra55do_ryf
* &null  njglobal                            300                                    MOM_input_om2_grid_iss36
* &null  nk                                  60                                     MOM_input_1deg_jra55do_ryf
* &null  nk                                  50                                     MOM_input_om2_grid_iss36
* &null  restore_salinity                    False                                  MOM_input_1deg_jra55do_ryf
* &null  restore_salinity                    True                                   MOM_input_om2_grid_iss36
* &null  salt_restore_file                                                          MOM_input_1deg_jra55do_ryf
* &null  salt_restore_file                   'salt_sfc_restore.nc'                  MOM_input_om2_grid_iss36
* &null  srestore_as_sflux                                                          MOM_input_1deg_jra55do_ryf
* &null  srestore_as_sflux                   True                                   MOM_input_om2_grid_iss36
* &null  temp_salt_init_vertical_remap_only                                         MOM_input_1deg_jra55do_ryf
* &null  temp_salt_init_vertical_remap_only  True                                   MOM_input_om2_grid_iss36
* &null  temp_salt_z_init_file               'WOA05_pottemp_salt.nc'                MOM_input_1deg_jra55do_ryf
* &null  temp_salt_z_init_file               'ocean_temp_salt.res.nc'               MOM_input_om2_grid_iss36
* &null  topo_file                           'ocean_topog_230424.nc'                MOM_input_1deg_jra55do_ryf
* &null  topo_file                           'topog.nc'                             MOM_input_om2_grid_iss36
* &null  tripolar_n                          False                                  MOM_input_1deg_jra55do_ryf
* &null  tripolar_n                          True                                   MOM_input_om2_grid_iss36
* &null  use_contemp_abssal                                                         MOM_input_1deg_jra55do_ryf
* &null  use_contemp_abssal                  False                                  MOM_input_om2_grid_iss36
* &null  z_init_file_ptemp_var               'PTEMP'                                MOM_input_1deg_jra55do_ryf
* &null  z_init_file_ptemp_var               'temp'                                 MOM_input_om2_grid_iss36
* &null  z_init_file_salt_var                'SALT'                                 MOM_input_1deg_jra55do_ryf
* &null  z_init_file_salt_var                'salt'                                 MOM_input_om2_grid_iss36

[aekiss]

looks good to me, thanks @dougiesquire