spack stack 1.6.0 unified-env install fails on Frontera with nemsio error

[benjamin-cash]

Subject line says most of it. I am trying to install the unified-dev template for spack-stack 1.6.0 on Frontera, and it is failing in the install step with an error related to nemsio and mpi. The output from the error is below:

==> [2024-06-03-12:23:56.964689] '/opt/apps/cmake/3.24.2/bin/cmake' '-G' 'Unix Makefiles' '-DCMAKE_INSTALL_PREFIX:STRING=/work2/02441/bcash/frontera/spack-stack/spack-stack-1.6.0/envs/unified-env/install/gcc/9.1.0/nemsio-2.5.4-qllwehu' '-DCMAKE_BUILD_TYPE:STRING=Release' '-DCMAKE_INTERPROCEDURAL_OPTIMIZATION:BOOL=OFF' '-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON' '-DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON' '-DCMAKE_INSTALL_RPATH:STRING=/work2/02441/bcash/frontera/spack-stack/spack-stack-1.6.0/envs/unified-env/install/gcc/9.1.0/nemsio-2.5.4-qllwehu/lib;/work2/02441/bcash/frontera/spack-stack/spack-stack-1.6.0/envs/unified-env/install/gcc/9.1.0/nemsio-2.5.4-qllwehu/lib64;/work2/02441/bcash/frontera/spack-stack/spack-stack-1.6.0/envs/unified-env/install/gcc/9.1.0/bacio-2.4.1-rrwobca/lib;/work2/02441/bcash/frontera/spack-stack/spack-stack-1.6.0/envs/unified-env/install/gcc/9.1.0/w3emc-2.10.0-iwv42sk/lib64' '-DCMAKE_PREFIX_PATH:STRING=/work2/02441/bcash/frontera/spack-stack/spack-stack-1.6.0/envs/unified-env/install/gcc/9.1.0/w3emc-2.10.0-iwv42sk;/work2/02441/bcash/frontera/spack-stack/spack-stack-1.6.0/envs/unified-env/install/gcc/9.1.0/bacio-2.4.1-rrwobca;/opt/intel/oneapi;/opt/apps/cmake/3.24.2' '-DENABLE_MPI:BOOL=ON' '-DCMAKE_Fortran_COMPILER:STRING=/opt/intel/oneapi/mpi/2021.9.0/bin/mpiifort' '/work2/02441/bcash/frontera/spack-stack-1.6.0/cache/build_stage/spack-stage-nemsio-2.5.4-qllwehulwj3247edu72d4ykgbpml6tbx/spack-src'
-- The Fortran compiler identification is GNU 9.1.0
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /opt/intel/oneapi/mpi/2021.9.0/bin/mpiifort - skipped
-- Found MPI_Fortran: /opt/intel/oneapi/mpi/2021.9.0/bin/mpiifort (found version "3.1")
-- Found MPI: TRUE (found version "3.1") found components: Fortran
-- Parallel NEMSIO Enabled
-- Found bacio: /work2/02441/bcash/frontera/spack-stack/spack-stack-1.6.0/envs/unified-env/install/gcc/9.1.0/bacio-2.4.1-rrwobca/lib/libbacio_4.a (found version "2.4.1")
-- Found bacio: /work2/02441/bcash/frontera/spack-stack/spack-stack-1.6.0/envs/unified-env/install/gcc/9.1.0/bacio-2.4.1-rrwobca/lib/libbacio_4.a (found version "2.4.1")
-- Found w3emc: /work2/02441/bcash/frontera/spack-stack/spack-stack-1.6.0/envs/unified-env/install/gcc/9.1.0/w3emc-2.10.0-iwv42sk/lib64/libw3emc_4.a (found version "2.10.0")
-- Configuring done
-- Generating done
-- Build files have been written to: /work2/02441/bcash/frontera/spack-stack-1.6.0/cache/build_stage/spack-stage-nemsio-2.5.4-qllwehulwj3247edu72d4ykgbpml6tbx/spack-build-qllwehu
==> nemsio: Executing phase: 'build'
==> [2024-06-03-12:23:59.000119] 'make' '-j3'
/opt/apps/cmake/3.24.2/bin/cmake -S/work2/02441/bcash/frontera/spack-stack-1.6.0/cache/build_stage/spack-stage-nemsio-2.5.4-qllwehulwj3247edu72d4ykgbpml6tbx/spack-src -B/work2/02441/bcash/frontera/spack-stack-1.6.0/cache/build_stage/spack-stage-nemsio-2.5.4-qllwehulwj3247edu72d4ykgbpml6tbx/spack-build-qllwehu --check-build-system CMakeFiles/Makefile.cmake 0
/opt/apps/cmake/3.24.2/bin/cmake -E cmake_progress_start /work2/02441/bcash/frontera/spack-stack-1.6.0/cache/build_stage/spack-stage-nemsio-2.5.4-qllwehulwj3247edu72d4ykgbpml6tbx/spack-build-qllwehu/CMakeFiles /work2/02441/bcash/frontera/spack-stack-1.6.0/cache/build_stage/spack-stage-nemsio-2.5.4-qllwehulwj3247edu72d4ykgbpml6tbx/spack-build-qllwehu//CMakeFiles/progress.marks
make  -f CMakeFiles/Makefile2 all
make[1]: Entering directory `/work2/02441/bcash/frontera/spack-stack-1.6.0/cache/build_stage/spack-stage-nemsio-2.5.4-qllwehulwj3247edu72d4ykgbpml6tbx/spack-build-qllwehu'
make  -f src/CMakeFiles/nemsio.dir/build.make src/CMakeFiles/nemsio.dir/depend
make[2]: Entering directory `/work2/02441/bcash/frontera/spack-stack-1.6.0/cache/build_stage/spack-stage-nemsio-2.5.4-qllwehulwj3247edu72d4ykgbpml6tbx/spack-build-qllwehu'
cd /work2/02441/bcash/frontera/spack-stack-1.6.0/cache/build_stage/spack-stage-nemsio-2.5.4-qllwehulwj3247edu72d4ykgbpml6tbx/spack-build-qllwehu && /opt/apps/cmake/3.24.2/bin/cmake -E cmake_depends "Unix Makefiles" /work2/02441/bcash/frontera/spack-stack-1.6.0/cache/build_stage/spack-stage-nemsio-2.5.4-qllwehulwj3247edu72d4ykgbpml6tbx/spack-src /work2/02441/bcash/frontera/spack-stack-1.6.0/cache/build_stage/spack-stage-nemsio-2.5.4-qllwehulwj3247edu72d4ykgbpml6tbx/spack-src/src /work2/02441/bcash/frontera/spack-stack-1.6.0/cache/build_stage/spack-stage-nemsio-2.5.4-qllwehulwj3247edu72d4ykgbpml6tbx/spack-build-qllwehu /work2/02441/bcash/frontera/spack-stack-1.6.0/cache/build_stage/spack-stage-nemsio-2.5.4-qllwehulwj3247edu72d4ykgbpml6tbx/spack-build-qllwehu/src /work2/02441/bcash/frontera/spack-stack-1.6.0/cache/build_stage/spack-stage-nemsio-2.5.4-qllwehulwj3247edu72d4ykgbpml6tbx/spack-build-qllwehu/src/CMakeFiles/nemsio.dir/DependInfo.cmake --color=
Scanning dependencies of target nemsio
make[2]: Leaving directory `/work2/02441/bcash/frontera/spack-stack-1.6.0/cache/build_stage/spack-stage-nemsio-2.5.4-qllwehulwj3247edu72d4ykgbpml6tbx/spack-build-qllwehu'
make  -f src/CMakeFiles/nemsio.dir/build.make src/CMakeFiles/nemsio.dir/build
make[2]: Entering directory `/work2/02441/bcash/frontera/spack-stack-1.6.0/cache/build_stage/spack-stage-nemsio-2.5.4-qllwehulwj3247edu72d4ykgbpml6tbx/spack-build-qllwehu'
[ 12%] Building Fortran object src/CMakeFiles/nemsio.dir/nemsio_openclose.f90.o
[ 12%] Building Fortran object src/CMakeFiles/nemsio.dir/nemsio_module_mpi.f90.o
cd /work2/02441/bcash/frontera/spack-stack-1.6.0/cache/build_stage/spack-stage-nemsio-2.5.4-qllwehulwj3247edu72d4ykgbpml6tbx/spack-build-qllwehu/src && /opt/intel/oneapi/mpi/2021.9.0/bin/mpiifort   -g -fconvert=big-endian  -O2 -Jinclude -c /work2/02441/bcash/frontera/spack-stack-1.6.0/cache/build_stage/spack-stage-nemsio-2.5.4-qllwehulwj3247edu72d4ykgbpml6tbx/spack-src/src/nemsio_openclose.f90 -o CMakeFiles/nemsio.dir/nemsio_openclose.f90.o
cd /work2/02441/bcash/frontera/spack-stack-1.6.0/cache/build_stage/spack-stage-nemsio-2.5.4-qllwehulwj3247edu72d4ykgbpml6tbx/spack-build-qllwehu/src && /opt/intel/oneapi/mpi/2021.9.0/bin/mpiifort   -g -fconvert=big-endian  -O2 -Jinclude -c /work2/02441/bcash/frontera/spack-stack-1.6.0/cache/build_stage/spack-stage-nemsio-2.5.4-qllwehulwj3247edu72d4ykgbpml6tbx/spack-src/src/nemsio_module_mpi.f90 -o CMakeFiles/nemsio.dir/nemsio_module_mpi.f90.o
f951: Fatal Error: Reading module 'mpi' at line 1 column 2: Unexpected EOF
compilation terminated.
make[2]: *** [src/CMakeFiles/nemsio.dir/nemsio_module_mpi.f90.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory `/work2/02441/bcash/frontera/spack-stack-1.6.0/cache/build_stage/spack-stage-nemsio-2.5.4-qllwehulwj3247edu72d4ykgbpml6tbx/spack-build-qllwehu'
make[1]: *** [src/CMakeFiles/nemsio.dir/all] Error 2
make[1]: Leaving directory `/work2/02441/bcash/frontera/spack-stack-1.6.0/cache/build_stage/spack-stage-nemsio-2.5.4-qllwehulwj3247edu72d4ykgbpml6tbx/spack-build-qllwehu'
make: *** [all] Error 2```

[benjamin-cash]

This same mpi error has cropped up now in a few more elements.

grep -i error log.install 
==> Error: ProcessError: Command exited with status 2:
3 errors found in build log:
     33    f951: Fatal Error: Reading module 'mpi' at line 1 column 2: Unexpected EOF
  >> 35    make[2]: *** [src/CMakeFiles/nemsio.dir/nemsio_module_mpi.f90.o] Error 1
  >> 38    make[1]: *** [src/CMakeFiles/nemsio.dir/all] Error 2
  >> 40    make: *** [all] Error 2
login3.frontera(1059)$ grep -i error log.install 
==> Error: ProcessError: Command exited with status 2:
3 errors found in build log:
     33    f951: Fatal Error: Reading module 'mpi' at line 1 column 2: Unexpected EOF
  >> 35    make[2]: *** [src/CMakeFiles/nemsio.dir/nemsio_module_mpi.f90.o] Error 1
  >> 38    make[1]: *** [src/CMakeFiles/nemsio.dir/all] Error 2
  >> 40    make: *** [all] Error 2
==> Error: ProcessError: Command exited with status 2:
5 errors found in build log:
     178    f951: Fatal Error: Reading module 'mpi' at line 1 column 2: Unexpected EOF
  >> 180    make[2]: *** [CMakeFiles/fms_r4_f.dir/mpp/mpp_data.F90.o] Error 1
     238    f951: Fatal Error: Reading module 'mpi' at line 1 column 2: Unexpected EOF
  >> 240    make[2]: *** [CMakeFiles/fms_r8_f.dir/mpp/mpp_data.F90.o] Error 1
  >> 243    make[1]: *** [CMakeFiles/fms_r4_f.dir/all] Error 2
  >> 246    make[1]: *** [CMakeFiles/fms_r8_f.dir/all] Error 2
  >> 248    make: *** [all] Error 2
==> Error: ProcessError: Command exited with status 2:
3 errors found in build log:
     174    f951: Fatal Error: Reading module 'mpi' at line 1 column 2: Unexpected EOF
  >> 176    make[2]: *** [src/gsibec/CMakeFiles/gsibec.dir/gsi/m_mpimod.F90.o] Error 1
  >> 179    make[1]: *** [src/gsibec/CMakeFiles/gsibec.dir/all] Error 2
  >> 181    make: *** [all] Error 2
==> Error: ProcessError: Command exited with status 2:
3 errors found in build log:
     244    f951: Fatal Error: Reading module 'mpi' at line 1 column 2: Unexpected EOF
  >> 246    make[2]: *** [src/flib/CMakeFiles/piof.dir/pio_kinds.F90.o] Error 1
  >> 248    make[1]: *** [src/flib/CMakeFiles/piof.dir/all] Error 2
  >> 250    make: *** [all] Error 2

[climbfuji]

Looks like MPI was built with a different compiler?

[benjamin-cash]

Thank @climbfuji - where would that kind of mismatch creep in from?

[climbfuji]

I don't know. I would check the concretizer log for different compilers being used (%gcc and %intel for example)

[benjamin-cash]

Hmm... I wonder if it only failed using gcc.

[benjamin-cash]

Update - it only failed with gcc. Not sure why that combination is an issue, but since I'm not planning on using gcc on Frontera I can go ahead and close this.