SchedMD SLURM image has two versions of MPI installed?

[Tristan-Kosciuch]

In the schedmd-v5-slurm-22-05-6-ubuntu-2004-lts family of images there are two versions of MPI installed. Running ompi_info shows Debian OpenMPI v4.0.3 installed under /usr. However:

module load openmpi
ompi_info

loads OpenMPI v4.1.5. Which is preferred to build applications with? I need to do mpirun tasks that involve multiple nodes. I noticed OpenMPI v4.1.5 doesn't show the --with-slurm flag was used when building MPI so wasn't sure if it would work here.

[cboneti]

Hi Tristan,

I am not sure either. Let me ask SchedMD.

[cboneti]

Hi Tristan, SchedMD is building 4.1.5 --with-slurm enabled by default, it is built against Slurm's PMI library, and so you should prefer using that one.

SchedMD also added that because of how debian's bash does its profile sourcing, you may need to source /etc/profile or /etc/profile.d/z00_lmod.sh to have module load available. This is needed if the bash session is not a login shell, which it looks like salloc and sbatch are not.

[Tristan-Kosciuch]

Thanks! Are there any necessary command line options to pass so slurm's PMI library is used? For example, this is a script I am submitting through sbatch but it fails when more than one node is used.

#!/bin/sh
source /etc/profile.d/z00_lmod.sh
module load openmpi
cd /home/$USER/my-program/
mpirun ./my-program

In the file /usr/local/etc/slurm/slurm.conf I noticed MpiDefault=none

[bsngardner]

MpiDefault=none (default) works with OpenMPI.

Your module command also works with my mpi test program. Not sure what could be wrong. Any error messages?

[Tristan-Kosciuch]

MpiDefault=none (default) works with OpenMPI.

Your module command also works with my mpi test program. Not sure what could be wrong. Any error messages?

I missed the message about "source /etc/profile or /etc/profile.d/z00_lmod.sh". The module command is working now.

[bsngardner]

A bit more detail in case you don't know:

# test.c
#include <stdio.h>
#include <mpi.h>

int main(int argc, char **argv)
{
   int node;

   MPI_Init(&argc, &argv);
   MPI_Comm_rank(MPI_COMM_WORLD, &node);

   printf("Hello World from Node %d\n", node);

   MPI_Finalize();
}

Tests run on a slurm-gcp cluster using schedmd-slurm-22-05-8-ubuntu-2004-lts.

4.1.5 built against slurm's libpmi

$ sbatch -N2 --wrap="bash --login -c 'module load openmpi && mpicc test.c -o test && srun ./test'"
$ sbatch -N2 --wrap="bash --login -c 'module load openmpi && mpicc test.c -o test && mpirun ./test'"
output file for both:
Hello World from Node 0
Hello World from Node 1

Both of these work and are (I think) essentially the same. OpenMPI's slurm integration allows mpirun to detect when it is inside a slurm job and behave correctly. mpirun might launch orted to handle communication when slurmd can do the job.

4.0.3 in from Debian

$ sbatch -N2 --wrap="mpicc test.c -o test && srun ./test"
output file:
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[bg1-main-c4-0:19515] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: bg1-main-c4-0: task 0: Exited with exit code 1
srun: error: bg1-main-n4-0: task 1: Exited with exit code 1
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[bg1-main-n4-0:11345] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!

This does not work due to lack of slurm pmi support.

$ sbatch -N2 --wrap="mpicc test.c -o test && mpirun ./test"

This appears to be working.

[bsngardner]

If I were submitting this as a script instead of a --wrap, I would use

#!/bin/bash
source /etc/profile/z00_lmod.sh
module load openmpi
mpicc -o test test.c
srun ./test

[Tristan-Kosciuch]

Thanks. The tests seem to work well but I get a response from each processor, not from each node. For example, using test.c with two c2-standard-16 compute nodes give this:

Hello World from Node 7
Hello World from Node 3
Hello World from Node 5
Hello World from Node 6
Hello World from Node 0
Hello World from Node 1
Hello World from Node 2
Hello World from Node 13
Hello World from Node 15
Hello World from Node 8
Hello World from Node 10
Hello World from Node 4
Hello World from Node 11
Hello World from Node 12
Hello World from Node 14
Hello World from Node 9

For my program, the error I'm running into is

--------------------------------------------------------------------------
ORTE has lost communication with a remote daemon.

  HNP daemon   : [[45494,0],0] on node slurmlustr-compute-ghpc-0
  Remote daemon: [[45494,0],3] on node slurmlustr-compute-ghpc-3

This is usually due to either a failure of the TCP network
connection to the node, or possibly an internal failure of
the daemon itself. We cannot recover from this failure, and
therefore will terminate the job.
--------------------------------------------------------------------------

There are no further error messages that I can see beyond what I posted above. The job will start but I believe it fails once the parallel MPI tasks begin. Also fyi this job is doing parallel file writes to a lustre (DDN exascaler) filesystem.