questions about custom grid

[SarahLu-NOAA]

I'm trying to set up a custom grid (over NE US) at Derecho. I have one successful run (/glade/work/clu/ufs/expt_dirs/RRFS_nys_12km_ver1). When I tweak WRTCMP parameters in config.yaml, it fails at the post step (/glade/work/clu/ufs/expt_dirs/RRFS_nys_12km). I also have one run failed at make_ics and make_lbcs with slightly modified WRTCMP parameters.

While these WRTCMP parameters are described in the documentation, it's not clear how to make these parameters consistent with the configuration parameters in task_make_grid. Apparently, I need some guidance on setting up grid points for the write-component grid. -Thanks -

[gspetro-NOAA]

Hi @SarahLu-NOAA ,

Just wanted to let you know that I passed this question along to one of our subject matter experts, and he should be responding to you soon!

Best,
Gillian

[SarahLu-NOAA]

@gspetro-NOAA
Many thanks, Gillian, for connecting me with @mkavulich

[mkavulich]

Sarah, in this case, the WRTCMP change causing MPI issues in UPP is a bit of a red herring: you've just happened to shrink your domain enough that over-decomposition is becoming a problem (too many processors for the domain size). You will need to assign fewer MPI tasks to run_post; you can do this by updating the rocoto: section in your config.yaml:

rocoto:
  tasks:
    taskgroups: '{{ ["parm/wflow/prep.yaml", "parm/wflow/coldstart.yaml", "parm/wflow/post.yaml",
       "parm/wflow/plot.yaml"]|include
      }}'
    metatask_run_ensemble:
      task_run_fcst_mem#mem#:
        walltime: 02:00:00
    metatask_run_ens_post:
       metatask_run_post_mem#mem#_all_fhrs:
         task_run_post_mem#mem#_f#fhr#:
           walltime: 00:20:00
           nnodes: 1
           ppn: 12

This will give UPP 12 processors instead of 48, which should be good enough to avoid the over-decomposition problem.

I would also recommend reducing the MPI processors assigned for the forecast step if you will be running such a small (in number of grid points) domain: even though the larger nodes of Derecho allow for the use of large numbers of processors, for very small domains you will actually see reduced performance by using all of these processors, with much longer runtimes due to additional halo communications needed for very small MPI patches. I would recommend reducing LAYOUT_X and LAYOUT_Y by half; and honestly going even smaller than that might be called for with such a small domain.

Furthermore, note the difference between the write component and the compute grid. The WRTCMP settings apply only to the write component (model output files); the compute grid (where the atmospheric integration takes place) is still the same, so reducing the size of the write component variables should typically be accompanied by reducing the ESGgrid_NX and ESGgrid_NY by a similar amount. Unfortunately there are no automated tools for comparing the compute grid to the write grid at this time, so this may require some trial-and-error to match the write grid to the compute grid.

Please let me know if you have further questions!

[SarahLu-NOAA]

@mkavulich
I modify config.yaml to use 12 processors for UPP and reduce LAYOUT_X and LAYOUT_Y by half. Just wonder whether I should reduce BLOCKSIZE as well?
I obviously don't quite understand how to set up configuration parameters for make_grid and WRTCMP. When I tweak the domain for NE US, I have some runs failed at post due to over-decomposition and some other runs failed at make_ics and make_lbcs. Another question I have is: how do I determine WRTCMP_nx and WRTCMP_ny. Can I set these same as ESGgrid_NX and ESGgrid_NY? Thanks.

[mkavulich]

BLOCKSIZE in general shouldn't be changed by a user on a known platform. I can't remember the exact specifics but it's a setting for the model only and I believe it's related to how memory is chunked at the processor level.

I assume any failures in make_ics or make_lbcs would also be related to over-decomposition in this case, you can change those with similar settings in the rocoto section under metatask_run_ensemble::

      task_make_ics_mem#mem#:
        nnodes: 1
        ppn: 4
      task_make_lbcs_mem#mem#:
        nnodes: 1
        ppn: 4

Another question I have is: how do I determine WRTCMP_nx and WRTCMP_ny. Can I set these same as ESGgrid_NX and ESGgrid_NY? Thanks.

You can set these to be the same, but due to differences in map projection type (the native grid is ESG..."extended schmidt gnomonic" vs the write component which is lambert conformal) the grids will never correspond 1-to-1. So you would typically make the compute grid a few gridpoints larger than your write grid, otherwise you may see some bands of undefined data on the edges of your output grid where the write grid extends beyond the compute grid. This isn't a problem for the model which is happy to write the files with undefined data, but it could cause some data processing issues downstream.