Optimization options

[hertneky]

I built the SRW v2.1 (and HPC-stack) on an unsupported platform (Borah: Boise State cluster) using intel/2021/2.0.2883 and mpi/2021/2.0.2883
I have successfully run on a small custom domain, testing various computing configurations. I have some questions about runtime performance for the forecast:

Borah has 48 processors per node. I notice if I run, say 6:ppn=48, the run time is MUCH slower compared to 6:ppn=24 with exclusive node use. I have run 553x355 domain 8-day run in 08:15 wall time, but trying to get it in under 7 hours using 6 nodes for an operational ensemble.

• Are there other compiler options that I could try building with that might help speed up runtime? (note that I had issues building HPC-stack with other options, but could revisit)
• Are there some optimization options I could try? I saw KMP_AFFINITY_RUN_FCST options could affect runtime speed and I read more about the available options, but not experienced enough to fully understand what might be best for my configuration.

[DavidHuber-NOAA]

@hertneky It's interesting that the runtime was so much slower when running 6:48. When run this way, OMP_NUM_THREADS_RUN_FCST needs to be set to 1 (it's 2 by default) so each PE runs on one core instead of 2. Did you try this? In my experience,, running with more instances and fewer threads tends to be faster. You might try setting that in your config file and rerunning setup.py. Note that this will use ~2x the memory on the node, so you might want to see if the 6:24 case was already bumping up against that limit. If you are running slurm for your job manager, you can check this with sacct -j <job number> -o "JobName,MaxRSS", where MaxRSS is the maximum memory used by a single PE. Multiply this number by 24 to get the maximum potential memory used on the entire node.

For KMP_AFFINITY_RUN_FCST, "scatter" is probably your best bet, which is the default, but you could also try "compact" if you want a second option.

For compiler options, did you create a cmake configuration file in /sorc/ufs-weather-model/cmake? If so, you could try compiling using FASTER in that cmake file:
set(FASTER ON CACHE BOOL "Enable AVX2 instruction set" FORCE)

[hertneky]

Thank you @DavidHuber-NOAA I just wanted to update you with my findings. The threads was indeed still 2. Setting to 1 sped up the run by about an hour. I also tried the FASTER compilation, which did not seem to help - it finished 5 mins earlier. I'm not sure I understand the RSS I am getting out. For this system the hardware is 48 cores per node and 192GB memory per node.
This was the info from the latest run (192-hr forecast):
JobName Start Elapsed NNodes NCPUS MaxRSS

---

run_fcst 2023-06-20T14:25:32 07:17:42 6 288
batch 2023-06-20T14:25:32 07:17:42 1 48 16492K
hydra_bst+ 2023-06-20T14:25:49 07:17:24 6 6 48585384K

Either way, this was helpful in getting it faster, although we were trying to get it under 7 hours and I don't see a way without increasing Nodes (limited for fitting into an ensemble).

[DavidHuber-NOAA]

Shucks, that is so close. You could try one other option to increase the time step. However, you run into the possibility of numerical instabilities, so I wouldn't change it by much and it needs to be a multiple of 3600s. You can change that in ush/predef_grid_params.yaml.

The MaxRSS indicates that the largest memory footprinton a node was ~48.5GB, so running single-threaded is probably fine assuming you have more than ~60GB/node.

[hertneky]

Thanks @DavidHuber-NOAA this was all very helpful. Good point about the time step. I had been playing with that before and I think it can be increased.