Compiling Kratos with MPI support

Compiling Kratos with MPI support

Running Kratos with MPI support requires compiling and enabling:

• Kratos' METIS Application, which provides an interface to the METIS graph partitioning library, to generate mesh partitions for MPI runs.
• Kratos' Trilinos Application, which adapts components of the Trilinos project, providing distributed-memory matrix and vector classes and linear solvers.

Once these dependencies are installed (either by downloading or by compiling them), building Kratos itself requires the following tweaks in the building script, as shown in the following snippets.

• Make Lapack and MPI libraries available:

# Load modules required for compilation
...
module load intel/oneapi2021/mkl/2021.1.1
module load openmpi
...

• Add TrilinosApplication and MetisApplication to the list of applications to compile:

export KRATOS_APPLICATIONS=
...
add_app ${KRATOS_APP_DIR}/TrilinosApplication
add_app ${KRATOS_APP_DIR}/MetisApplication

• Enable MPI and OpenMP parallelization, and hint cmake the location of Trilinos and Metis libraries (highly dependent on the system setup):

cmake -H"${KRATOS_SOURCE}" -B"${KRATOS_BUILD}" \
...
-DUSE_MPI=ON \
-DKRATOS_SHARED_MEMORY_PARALLELIZATION="OpenMP" \
-DTRILINOS_ROOT="${HOME}/Projects/Trilinos/build" \
-DMETIS_ROOT_DIR="${HOME}/Projects/METIS/build" \

A complete working script can be found below. Also, don't forget to add the libraries path to the environment variables before executing Kratos.

Getting dependencies from packages

The easiest way to get the required libraries is by the Linux distribution's package manager.

For example, for Ubuntu:

sudo apt install libmetis-dev
sudo apt install trilinos-all-dev

will install all the Trilinos packages. To install only the Trilinos packages required by the TrilinosApplication:

sudo apt install \
        libtrilinos-amesos-dev \
        libtrilinos-amesos2-dev \
        libtrilinos-aztecoo-dev \
        libtrilinos-epetra-dev \
        libtrilinos-epetraext-dev \
        libtrilinos-ifpack-dev \
        libtrilinos-ml-dev \
        libtrilinos-teuchos-dev \
        libtrilinos-tpetra-dev \
        libtrilinos-kokkos-dev \
        libtrilinos-kokkos-kernels-dev \
        libtrilinos-shylu-dev

Compiling dependencies

In some cases (e.g., no admin permissions, no packages or modules present, outdated versions) there may be necessary to build the libraries. This is a set of scrips for compiling GKlib, METIS and Trilinos libraries and their dependencies. Following the building instructions of their documentation is not enough for compiling them successfully for Kratos, as at the moment there are some workarounds the need to be applied.

The following scripts are just guidelines, and most likely they need to be adapted to each individual setup. For each package, a specific commit hash tag is cloned from its repository in GitHub, to allow reproducibility of the compilation.

We are assuming that they all are cloned and built in a ROOT directory, for example:

export ROOT=${HOME}/Projects

Tested with cmake v3.24, gcc v10.2

NOTE: As of July 6, 2022, there are a couple of files requiring a C++ standard >= 14 to compile. We won't fix this issue as Kratos is transitioning to C++17 at the moment, which will solve the problem automatically.

Step 1. Getting and building GKlib

GKlib is a dependency of METIS, so it has to be built first. The key step for building is to add the compilation flag below to avoid compilation errors in some systems.

#! /usr/bin/sh

ROOT=${HOME}/Projects

###############################################
# GKlib
# - Last tested on July 5th, 2022
COMMIT=a7f8172703cf6e999dd0710eb279bba513da4fec

# Clone
cd ${ROOT}
git clone --depth 1 [email protected]:KarypisLab/GKlib.git
cd GKlib
git checkout ${COMMIT}

# Build
cmake \
-DBUILD_SHARED_LIBS=ON \
-DCMAKE_INSTALL_PREFIX=${ROOT}/GKlib/build \
-DCMAKE_C_FLAGS="${CMAKE_C_FLAGS} -D_POSIX_C_SOURCE=199309L" \
.
make install

Step 2. Getting and building METIS

Building METIS requires patching CMakeLists.txt file.

#! /usr/bin/sh

ROOT=${HOME}/Projects

###############################################
# METIS
# - Last tested on July 5th, 2022
COMMIT=94c03a6e2d1860128c2d0675cbbb86ad4f261256

# Clone and checkout that specific commit
cd ${ROOT}
git clone --depth 1 [email protected]:KarypisLab/METIS.git 
cd METIS
git checkout ${COMMIT}

# Compile
cd libmetis
echo "include_directories(.)" > CMakeLists.txt
echo "" >> CMakeLists.txt
echo "file(GLOB metis_sources *.c)" >> CMakeLists.txt
echo "" >> CMakeLists.txt
echo "add_library(metis \${METIS_LIBRARY_TYPE} \${metis_sources})" >> CMakeLists.txt
echo "target_link_libraries(metis PUBLIC GKlib)" >> CMakeLists.txt
echo "" >> CMakeLists.txt
echo "if(METIS_INSTALL)" >> CMakeLists.txt
echo "  install(TARGETS metis" >> CMakeLists.txt
echo "    LIBRARY DESTINATION lib" >> CMakeLists.txt
echo "    RUNTIME DESTINATION lib" >> CMakeLists.txt
echo "    ARCHIVE DESTINATION lib)" >> CMakeLists.txt
echo "endif()" >> CMakeLists.txt
cd ..
make config shared=1 prefix=${ROOT}/METIS/build gklib_path=${ROOT}/GKlib/build
make install

Step 3. Getting and building Trilinos

The following script builds Trilinos with only the packages required by the TrilinosApplication. Building requires LAPACK and an installation of MPI (OpenMPI in this case.) Intel's MKL is only used to provide BLAS and LAPACK.

# Trilinos
# - Last tested on July 4th, 2022
# - Commit 9c03d9b

# Clone
git clone [email protected]:trilinos/Trilinos.git

# Build
module load intel/oneapi2021/mkl/2021.1.1  # provides LAPACK
module load openmpi  # provides MPI

cd Trilinos
mkdir -p build
cd build
rm -rf CMakeCache.txt CMakeFiles/ cmake_install.cmake

export MAKEFLAGS=
cmake \
-D CMAKE_INSTALL_PREFIX=$WORK/Trilinos/build \
-D CMAKE_BUILD_TYPE="RELWITHDEBINFO" \
-D TPL_ENABLE_MPI=ON \
-D BUILD_SHARED_LIBS=ON \
-D Trilinos_ENABLE_Epetra=ON \
-D Trilinos_ENABLE_EpetraExt=ON \
-D Trilinos_ENABLE_Triutils=ON \
-D Trilinos_ENABLE_AztecOO=ON \
-D Trilinos_ENABLE_Ifpack=ON \
-D Trilinos_ENABLE_ML=ON \
-D Trilinos_ENABLE_Amesos=ON \
-D Trilinos_ENABLE_Amesos2=ON \
-D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES=OFF \
-D TPL_ENABLE_MKL=OFF \
-D BLAS_LIBRARY_DIRS="${MKLROOT}/lib/intel64" \
-D BLAS_LIBRARY_NAMES="mkl_rt" \
-D LAPACK_LIBRARY_DIRS="${MKLROOT}/lib/intel64" \
-D LAPACK_LIBRARY_NAMES="mkl_rt" \
$WORK/Trilinos
make -j 6 install

Compiling Kratos

This is a working script that build Kratos using the Trilinos and METIS libraries compiled as shown in the previous section.

#!/bin/bash

# Function to add apps
add_app () {
        export KRATOS_APPLICATIONS="${KRATOS_APPLICATIONS}$1;"
}

# Load modules required for compilation
module purge
module load boost
module load python
module load git
module load intel/oneapi2021/mkl/2021.1.1
module load openmpi
module switch gcc gcc/10.2.0

# Set branch
export KRATOS_BRANCH="mpi-gcc"

# Set variables
export KRATOS_SOURCE="${KRATOS_SOURCE:-"$( cd "$(dirname "$0")" ; pwd -P )"/..}"
export KRATOS_BIN="${KRATOS_SOURCE}/bin/${KRATOS_BRANCH}"
export KRATOS_BUILD="${KRATOS_SOURCE}/build/${KRATOS_BRANCH}"
export KRATOS_APP_DIR="${KRATOS_SOURCE}/applications"
export KRATOS_INSTALL_PYTHON_USING_LINKS=ON

# Set basic configuration
export KRATOS_BUILD_TYPE=${KRATOS_BUILD_TYPE:-"RelWithDebInfo"}

# Set applications to compile
export KRATOS_APPLICATIONS=
add_app ${KRATOS_APP_DIR}/FluidDynamicApplication
add_app ${KRATOS_APP_DIR}/TrilinosApplication
add_app ${KRATOS_APP_DIR}/MetisApplication

# Clean cmake cache
rm -rf "${KRATOS_BUILD}/cmake_install.cmake"
rm -rf "${KRATOS_BUILD}/CMakeCache.txt"
rm -rf "${KRATOS_BUILD}/CMakeFiles"

# Configure
cmake -H"${KRATOS_SOURCE}" -B"${KRATOS_BUILD}" -GNinja \
-DUSE_MPI=ON \
-DKRATOS_SHARED_MEMORY_PARALLELIZATION="OpenMP" \
-DKRATOS_BUILD_TESTING=OFF \
-DINSTALL_RUNKRATOS=OFF \
-DCMAKE_INSTALL_PREFIX="${KRATOS_BIN}" \
-DTRILINOS_ROOT="${HOME}/Projects/Trilinos/build" \
-DMETIS_ROOT_DIR="${HOME}/Projects/METIS/build" \

# Buid
cmake --build "${KRATOS_BUILD}" --target install -- -j 20

# Set current install
ln -sfn ${KRATOS_BRANCH} ${KRATOS_SOURCE}/bin/current

Update library path environment variables

This step in not necessary for compiling, but for running. Before executing Kratos, make sure that the built libraries are in the LD_LIBRARY_PATH environment variable, so the system can find the libraries:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$WORK/Trilinos/build/lib
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$WORK/GKlib/build/lib
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$WORK/METIS/build/lib

Also, don't forget to specify MPI as the parallel scheme when setting the parameters of the project in ProjectParameters.json:

{
    "problem_data": {
        ...
        "parallel_type": "MPI",
        ...
    },
    ...
}