Performance degradation over time on Frontier

[pgrete]

Here's a representative performance plot on Frontier (1068 nodes) from AthenaPK for a simulation SMR (thus constant workload per cycle) showing wall seconds per cycle versus cycle.

Restart within a single job "resets" the performance, so it's nothing general wrt individual node performance.

I'm reporting here because (as discussed during sync) @bprather has also seen this (on smaller scales, like 8 nodes), so it's likely related to Parthenon and/or Frontier and not AthenaPK itself.

For reference:
AthenaPK Dec24 is f8497c5 (with associated submodule)
AthenaPK Oct23 is 3ce0a88

Stack Mid24 is

module restore
module load cpe/24.07 PrgEnv-amd cray-mpich/8.1.30 craype-accel-amd-gfx90a amd/6.2.0 rocm/6.2.0
module load cmake cray-python
module unload darshan-runtime

export LD_LIBRARY_PATH=${CRAY_LD_LIBRARY_PATH}:${LD_LIBRARY_PATH}
export MPICH_GPU_SUPPORT_ENABLED=1
export MPICH_GPU_IPC_CACHE_MAX_SIZE=1000
export MPICH_MPIIO_HINTS="*:romio_cb_write=disable"
export FI_MR_CACHE_MONITOR=disable

Stack Dec24 is

module load PrgEnv-cray
module load cpe/23.12
module load craype-accel-amd-gfx90a
module load rocm/5.7.1
module load cray-mpich
module load cce/17.0.0
module load cray-hdf5-parallel/1.14.3.1

export MPICH_GPU_SUPPORT_ENABLED=1
export FI_CXI_RX_MATCH_MODE=software
export MPICH_MPIIO_HINTS="*:romio_cb_write=disable" 

Next year todos:

• smaller reproducer (ideally in Parthenon)
• check if this tied to
  • is environment vars
  • cray versus amd module
• potentially bisect Parthenon between known good and current version

[BenWibking]

Does it reproduce with the older cray-mpich and the new binary?