HR CAM7 simulations

[cecilehannay]

We are planning a series of CAM7 simulations varying both horizontal and vertical resolutions. Specifically, we will perform AMIP simulations with observed sea surface temperature (SST) data and SST+4K at 1° and 0.25° resolutions, with both the CAM6 and CAM7 grids (i.e, with 32, 58, 93 vertical levels).

These simulations will be used to investigate the impact of both vertical and horizontal resolution on climate sensitivity, regional precipitation patterns, gravity waves, boundary layer aerosol-cloud interactions and hurricane activity

• Run type: AMIP type for years 2000-2010.
• Tag: we will use one of the latest tags. Specific configuration is to TBD
• Output: We are collecting the request for output

Res	Period	Levels	Experiment	Physics	Comments
1 deg	2000-2010	L32	SST	cam7
			SST+4K	cam7
		L58	SST	cam7
			SST+4K	cam7
		L93	SST	cam7
			SST+4K	cam7
0.25 deg	2000-2010	L32	SST	cam7
			SST+4K	cam7
		L58	SST	cam7
			SST+4K	cam7
		L93	SST	cam7
			SST+4K	cam7
0.25 deg	2000-2005	L58	SST	cam5	5 or 10 years
			SST+4K	cam5
0.25 deg	2000-2005	L58	SST	prog mom off	Prognostics momentum disabled: 5 or 10 years?
			SST+4K	prog mom off	Do we want this too?

@JulioTBacmeister, @adamrher, @islasimpson, @swrneale, @PeterHjortLauritzen, @cmcclus, @bstephens82

[cecilehannay]

Output

• monthly

fincl1 = 'ACTNI', 'ACTNL', 'ACTREI', 'ACTREL', 'AODDUST', 'AODVIS', 'AODVISdn','BURDENBC', 'BURDENDUST', 'BURDENPOM', 'BURDENSEASALT', 
'BURDENSO4', 'BURDENSOA', 'CAPE', 'CCN3', 'CDNUMC', 'CH4', 'CLDHGH', 'CLDICE', 'CLDLIQ', 'CLDLOW', 'CLDMED', 'CLDTOT', 'CLOUD', 'CMFMC_DP', 
'CT_H2O', 'DCQ', 'DQCORE', 'DTCOND', 'DTCORE', 'DTV', 'EVAPPREC', 'EVAPSNOW', 'FCTI', 'FCTL', 'FICE', 'FLDS', 'FLNS', 'FLNSC', 'FLNT', 'FLNTC', 'FLUT', 
'FREQZM', 'FSDS', 'FSDSC', 'FSNS', 'FSNSC', 'FSNT', 'FSNTC', 'FSNTOA', 'ICEFRAC', 'LANDFRAC', 'LHFLX', 'LWCF', 'MPDICE', 'MPDLIQ', 'MPDQ', 'MPDT', 
'OCNFRAC', 'OMEGA', 'OMEGA500', 'PBLH', 'PHIS', 'PINT', 'PMID', 'PRECC', 'PRECL', 'PRECSC', 'PRECSL', 'PRECT', 'PS', 'PSL', 'PTEQ', 'PTTEND', 'Q', 
'QFLX', 'QRL', 'QRS', 'QTGW', 'RCMTEND_CLUBB', 'RELHUM', 'RVMTEND_CLUBB', 'SHFLX', 'SOLIN', 'SST', 'STEND_CLUBB', 'SWCF', 
'T', 'TAUX', 'TAUY', 'TFIX', 'TGCLDIWP', 'TGCLDLWP', 'TMQ', 'TREFHT', 'TS', 'TTGW', 'U', 'U10', 'UBOT', 'UTGWORO', 'UTGW_TOTAL', 
'V', 'VBOT', 'VTGWORO', 'VTGW_TOTAL', 'WPRTP_CLUBB', 'WPTHLP_CLUBB', 'Z3', 'ZMDQ', 'ZMDT', 'N2O', 'CO2','CFC11','CFC12', 
'AODVISdn','CCN3', 'CDNUMC', 'H2O', 'NUMICE', 'NUMLIQ','OMEGA500'

daily

fincl3 = 'PRECT', 'PRECC', 'FLUT', 'U850', 'U200', 'V850', 'V200', 'OMEGA500', 'TS', 'SST', 'PSL'

-3 hourly

fincl4 =  'PRECC','PRECL'

monthly

fincl5 = 'Uzm','Vzm','Wzm','THzm', 'VTHzm','WTHzm','UVzm','UWzm'
phys_grid_ctem_nfreq=-6
phys_grid_ctem_zm_nbas=120
phys_grid_ctem_za_nlat=90

We are collecting output. Please indicate the requested output (for instance:

• @bstephens82: add tempest extreme
• @cmcclus: add output for southern ocean.

[swrneale]

For monthly mean maybe a few Tropopause diags. 'TROP_P','TROP_T','TROP_Z'. And a few convective diags. 'LCL','PLCL','LEL'. These are all 2D so not big.

[bstephens82]

• @bstephens82: add tempest extreme

For TempestExtremes, @JulioTBacmeister and I would request 3-hourly instantaneous output of PSL, U850, V850, Z300 and Z500. These are all 2d fields so hopefully that's fine. I believe Julio will request some precip variables.

I will also request some additional variables to analyze upgradient fluxes from the new prognostic momentum flux code. I will try to get the details for this in by tomorrow 9/10/24.

[bstephens82]

@cecilehannay the new diagnostics I would like to request require a sourcemod version of clubb_intr.F90. Are you already going to be using a sourcemod version of that file? Or can I just base my changes on clubb_intr.F90 from a recent tag?

[cecilehannay]

@bstephens82: I don't use a modified clubb_intr.F90. So please base your changes on recent tags.

[cecilehannay]

Simone requested: AODDUSTdn

[PeterHjortLauritzen]

If the compute hours are coming out of AMWG CSL allocation then we need to be a little careful about the high res simulations unless we don't do ne240 simulations in FY25. If we don't do ne240 simulations then we can do approximately 6 10 year simulations at 1/4 degree (and no core hours to do high res CAM7 development in FY25).

Here are the numbers from the FY25 AMWG allocation request:

[adamrher]

@cecilehannay I have about 7M core hours I can offer for this work, which expires on Halloween. The project code is CESM0024 and I can go ahead and add you to that.

[shieldsca]

@cecilehannay For AR tracking, it would be great to have 6 -hourly instantaneous uIVT, vIVT, TMQ, PS, U850, V850

[JulioTBacmeister]

@cecilehannay It would be great to have PRECC and PRECL at 3-hourly frequency - both instantaneous and time-averaged

Also @bstephens82 , I think all of your upgradient and other CLUBB diagnostics should be written on the native SE horizontal grid to avoid confusion introduced by remapping. As far as the other fields I am agnostic about whether they should be regridded or not. TempestExtremes can handle either latlon or SE grids, right?

@shieldsca do you have a preference for latlon vs SE-grid outputs?

[bstephens82]

Also @bstephens82 , I think all of your upgradient and other CLUBB diagnostics should be written on the native SE horizontal grid to avoid confusion introduced by remapping. As far as the other fields I am agnostic about whether they should be regridded or not. TempestExtremes can handle either latlon or SE grids, right?

Yes, all of the diagnostics I'm requesting, for both tempest extremes and upgradient flux analysis, should be on the native unstructured grid.

[shieldsca]

AR tracking variables can be either, we can add regridding to the workflow during the detection process. Native grid is totally fine.

[cecilehannay]

This is started to be a bit confusing as people wants some output instantaneous versus average, interpolated versus native.

I gathered all the info in a google doc instead.
https://docs.google.com/document/d/1T9T8Axz1L58G0YB__QNXsx5u4rpUHEhXdFFcVQP6uJY/edit?usp=sharing

I will tag your names.
Please check you have what you want.

[PeterHjortLauritzen]

Can you set

se_statefreq=576

in the 1/4 degree runs and

se_statefreq=144

in the 1 degree runs (I would like to monitor max/min winds and other state variables).

Thanks

[bstephens82]

Hi @cecilehannay, I have created a SourceMod version of clubb_intr.F90 that will output some additional diagnostics for analyzing upgradient fluxes. You can find it at

/glade/work/stepheba/for_cecile/clubb_intr.F90

To go along with this file, could you please add the following 6-hourly instantaneous output (I included the user_nl_cam lines for the clubb_history also to make it easier for you):

<fincl stream>='um:I','vm:I','upwp:I','vpwp:I','up2_tp:I','vp2_tp:I','UPWP_UG_CLUBB:I','VPWP_UG_CLUBB:I','UPWP_UG_CT_CLUBB:I','VPWP_UG_CT_CLUBB:I','DUDZ_CLUBB:I','DVDZ_CLUBB:I','UPWP_UG_MEAN_LEV:I','UPWP_UG_MIN_LEV:I','UPWP_UG_MAX_LEV:I','VPWP_UG_MEAN_LEV:I','VPWP_UG_MIN_LEV:I','VPWP_UG_MAX_LEV:I','UPWP_UG_MEAN_HGT:I','VPWP_UG_MEAN_HGT:I'

clubb_history=.true.

clubb_vars_zt='um','vm'
clubb_vars_zm='upwp','vpwp','up2_tp','vp2_tp'

[adamrher]

there's a history_clubb too!? omg ...

[cecilehannay]

Yes, history_clubb = true allows to automatically output a set of variables on the h0 file.

Similarly we have other flags

• history_amwg => the default amwg variables
• history_aerosol
• history_chemistry
• history_dust
• history_waccm
• ...

I had no clue that there was a clubb_history flag with a very different behavior.

[bstephens82]

Yes, it's confusing. Might even be worth renaming someday...

[cecilehannay]

I noticed that the default has been to set ndens=1 for some tapes.
-'1' implies output real values are 8-byte
-'2' implies output real values are 4-byte.

Do we really need ndens=1 on some tape or was it just an oversight?
For ne30, it didn't matter too much but not ne120, saving with ndens=2 could save quite a bit of space.

[jedwards4b]

I agree that it should be ndens=2, for which tapes is this set to 1?

[cecilehannay]

@wwieder, @ekluzek, @slevis-lmwg
We are trying to set AMIP simulation at ne120.

/glade/campaign/cesm/cesmdata/cseg/runs/cesm2_0/f.e30_cam6_4_032.FMTHIST.ne120_L93.001_t1

We need to set a flanduse_timeseries. Do you have something that we could use? If not, is it possible to create a file?

As I am testing size of input, I tried to use the only ne120 file I found (just for testing purpose).

flanduse_timeseries = '/glade/campaign/cesm/cesmdata/cseg/inputdata/lnd/clm2/surfdata_esmf/ctsm5.2.0/landuse.timeseries_ne120np4_SSP5-8.5_78_CMIP6_1850-2100_c230530.nc'

but it doesn't seem compatible with the tag I used.

[cecilehannay]

Thanks. I tried this and it solves my issue (although now I have other issues but not in the land).

Now it is not clear why I shoudl not run transient land use. Our previous FMTHIST used transient land use. Is it incorrect?

[wwieder]

for a 10 year smoke test I don't think land use change is very important to consider, and this let's you address other non-land issues 😄 .

The other thing I'll note is that your surface dataset will also have the issues with soil texture we were discussing in the co-chair meeting this week. I think the newer file, below, should work for what you need since you're running with a stub glacier model? Is this accurate @billsacks?
/glade/campaign/cesm/cesmdata/inputdata/lnd/clm2/surfdata_esmf/ctsm5.3.0/surfdata_ne120np4.pg3_hist_1850_78pfts_c240908.nc.

Finally, @cecilehannay raises a good point here @ekluzek and @slevis-lmwg; we seem to have a land use time series for the mpas120 grid, but not for ne120. I'm assuming this was intentional, but don't want to assume?

[billsacks]

The other thing I'll note is that your surface dataset will also have the issues with soil texture we were discussing in the co-chair meeting this week. I think the newer file, below, should work for what you need since you're running with a stub glacier model? Is this accurate @billsacks?
/glade/campaign/cesm/cesmdata/inputdata/lnd/clm2/surfdata_esmf/ctsm5.3.0/surfdata_ne120np4.pg3_hist_1850_78pfts_c240908.nc.

Yes, if you're running with SGLC, then I think you can use these new surface datasets in pre-CTSM5.3.0 code.

[adamrher]

@billsacks if that doesn't work, can I just run the ctsm5.2 mksurf tools with a two line change to resolve the soil texture issue: https://github.com/slevis-lmwg/ctsm/pull/9/files#r1747755326 ?

[billsacks]

if that doesn't work, can I just run the ctsm5.2 mksurf tools with a two line change to resolve the soil texture issue: https://github.com/slevis-lmwg/ctsm/pull/9/files#r1747755326 ?

Yes, you can fix the issue by running with the changes in that small PR. You'll need most of the other changes in that PR, and you might as well just bring them all in - e.g., there are some declarations that you need.

[cecilehannay]

Crash in the ne120_L93.
After some issues, we were able to pass the issue with initialization.
However, the simulation is now crashing with a non reproducible error after 16 days in one occurrence and after 17 days in another occurrence.

There is no useful error messages in the log files (at least not useful to me).
The log files are in:

/glade/derecho/scratch/hannay/f.e30_cam6_4_032.FMTHIST.ne120_L93.001_t0

In one case the error is:

[dec1166.hsn.de.hpc.ucar.edu](http://dec1166.hsn.de.hpc.ucar.edu/) 1:  Opened existing file 
[dec1166.hsn.de.hpc.ucar.edu](http://dec1166.hsn.de.hpc.ucar.edu/) 1:  /glade/campaign/cesm/cesmdata/inputdata/atm/waccm/lb/LBC_17500116-20150116_CMIP
[dec1166.hsn.de.hpc.ucar.edu](http://dec1166.hsn.de.hpc.ucar.edu/) 1:  [6_0p5degLat_c180905.nc](http://6_0p5deglat_c180905.nc/)         294
[dec1166.hsn.de.hpc.ucar.edu](http://dec1166.hsn.de.hpc.ucar.edu/) 0:   sysmem size=3403.4 MB rss=1824.1 MB share=89.6 MB text=47.7 MB datastack=0.0 MB
[dec1166.hsn.de.hpc.ucar.edu](http://dec1166.hsn.de.hpc.ucar.edu/) 0:   sysmem size=3419.5 MB rss=1833.2 MB share=89.4 MB text=47.7 MB datastack=0.0 MB
[dec1610.hsn.de.hpc.ucar.edu](http://dec1610.hsn.de.hpc.ucar.edu/): rank -1 died from signal 9
[dec1166.hsn.de.hpc.ucar.edu](http://dec1166.hsn.de.hpc.ucar.edu/): rank 0 died from signal 15

In the other, I got the error:

[dec0246.hsn.de.hpc.ucar.edu](http://dec0246.hsn.de.hpc.ucar.edu/) 1410:   wetdapa: negative value           12          81 -5.679344388609238E-037
[dec0246.hsn.de.hpc.ucar.edu](http://dec0246.hsn.de.hpc.ucar.edu/) 1410:  -5.679344388609238E-037  0.000000000000000E+000  0.000000000000000E+000
[dec0246.hsn.de.hpc.ucar.edu](http://dec0246.hsn.de.hpc.ucar.edu/) 1410:  -0.567934438860924       0.000000000000000E+000
[dec0246.hsn.de.hpc.ucar.edu](http://dec0246.hsn.de.hpc.ucar.edu/): rank -1 died from signal 9
[dec0225.hsn.de.hpc.ucar.edu](http://dec0225.hsn.de.hpc.ucar.edu/): rank 0 died from signal 15

Could this be a memory leak?
If I remember correctly in August, there was a memory leak in PUMAS. Was it fixed? (@adamrher, @jedwards4b)

NB: Note that this test is not doing any output yet. I was just trying to see if I could run for a one month.

[cecilehannay]

Thanks, @adamrher. I have not changed anything with the land datasets yet except to set the

flanduse_timeseries = ' '  

I can use the ne120 finidat:

finidat = /glade/campaign/cesm/cesmdata/inputdata/lnd/clm2/initdata_map/clmi.F2000.2000-01-01.ne120pg3_mt13_simyr2000_c200728.nc

Should I also set the ctsm5.3 fsurfdat?

 fsurdat = '/glade/campaign/cesm/cesmdata/inputdata/lnd/clm2/surfdata_esmf/ctsm5.3.0/surfdata_ne120np4.pg3_hist_1850_78pfts_c240908.nc'

One more question: If the error is due to negative deposition fluxes, is it expected that it is not reproducible? The run crashed after a very different number of timesteps.

[adamrher]

Jim and Cecile are rightly skeptical of my attribution of the crashes to the negative values in wetdepa -- those are just warnings as I can't find any endrun associated with those write statements in the code. Further, the negative values are not even associated with the surface, with bad levels consistently around level 80. Sorry for being misleading.

I checked the memory usage in these two runs via qhist, and they are about 220GB per node. Seems high @jedwards4b?

[jedwards4b]

Yes I do think it's running out of memory. Do we have a fix yet for the clubb leak that we detected?

[adamrher]

You mean PUMAS ESCOMP/CAM#1136 ?

Julio ran this configuration not long ago: /glade/work/juliob/my_cam_development/cases/cDEV_ne120pg3_F2000_L93_x01/. He's running with NTASKS=5120, so I'd suggest we start with that.

[jedwards4b]

He's removed the run directory - so no help. And yes that's the issue I was referring to, thanks.

[slevis-lmwg]

Not to distract, but to answer these earlier questions:

Should I also set the ctsm5.3 fsurfdat?

 fsurdat = '/glade/campaign/cesm/cesmdata/inputdata/lnd/clm2/surfdata_esmf/ctsm5.3.0/surfdata_ne120np4.pg3_hist_1850_78pfts_c240908.nc'

@cecilehannay based on my reading of previous posts, yes, use the ctsm5.3.0 surface dataset.

we seem to have a land use time series for the mpas120 grid, but not for ne120. I'm assuming this was intentional, but don't want to assume?

@wwieder this was intentional as shown in the "grids" spreadsheet.

[wwieder]

I would not worry about land use for a 10 year run, especially for a proof of capability simulation.

[cecilehannay]

To clarify: This study is much more than just a demonstration of capability.

We aim to assess the impact of both vertical and horizontal resolution on climate sensitivity, regional precipitation patterns, gravity waves, boundary layer aerosol-cloud interactions, and hurricane activity. It will be a comprehensive study that many people will contribute to

[adamrher]

I am happy to generate a transient landuse_timeseries file for a 2000-2010 folks. It's quite easy to do with the new ctsm tools.

[ekluzek]

Having @adamrher create a dataset in the short term makes sense to me. In the longer term it does sound like you still want a 1979-2026 landuse timeseries file for ne120 that we support as a standard dataset in LMWG. I just want to make sure I have that right.

The issue in CTSM is here:

ESCOMP/CTSM#2785

[adamrher]

Yes, that's what we would prefer longer term. I've added 1979-2026 to the ne120pg3 entry in the cesm3 grids doc. I also added a note "1979-2026 landuse_timeseries file may be challenging to support due to storage limitations" ... is that a fair disclaimer?

[jedwards4b]

@cecilehannay try changing:
NTASKS=6144

[jedwards4b]

@cecilehannay I now have a running version in /glade/derecho/scratch/jedwards/PFS_P6144.ne120pg3_ne120pg3_mt13.FMTHIST.derecho_intel.20240920_144335_cl7c5y a number of changes to user_nl_clm were required.

[cecilehannay]

We are locking the discussion and starting a new one here: #587