Skip to content

Building

Jump to bottom
Luc Berger edited this page May 19, 2023 · 23 revisions

Building

KokkosKernels exists as a stand-alone sub-package of the Kokkos Ecosystem, as well as a package within the Trilinos Project

To use KokkosKernels as a stand-alone library requires use of the standalone CMake system. To use KokkosKernels as a package within Trilinos requires use of the CMake + TriBITS build system.

General Requirements:

  • Compatible versions of Kokkos and KokkosKernels cloned or downloaded from https://github.com
  • Supported compiler and computing hardware - see Kokkos' README for what is currently tested.
  • CUDA builds require use of the nvcc_wrapper script provided by Kokkos, unless using Clang-CUDA

Basic steps for building stand-alone KokkosKernels:

  1. Create a build directory <BUILD_DIR> (different from source and install locations).
> mkdir <BUILD_DIR>
cd <BUILD_DIR>
  1. Run cmake ${SOURCE_DIR} <...> where SOURCE_DIR is the location of the Kokkos Kernels source. <...> is a list of CMake options given as `-D{OPTION}={VALUE}'
  2. Build and install the library, depending on the generator (make is default):
> make install

or

> ninja install

To use the Ninja build system add -G Ninja.

A full listing of CMake options is given below. Another way to get a list of options and documentation is to the use ccmake utility.

ccmake <SOURCE_DIR>

which brings up a user interface listing all the options, their default values, and associated documentation.

Sample CMake

Below is a list of example CMake configurations. Kokkos Kernels requires first building Kokkos (or including it as a subproject). To see a full list of options for building Kokkos, see BUILD

OpenMP backend, g++ compiler, Intel Skylake architecture:

First install Kokkos:

> cmake \
  -DCMAKE_CXX_COMPILER=g++ \
  -DCMAKE_INSTALL_PREFIX=${HOME}/kokkos-install \
  -DKokkos_ENABLE_OPENMP=ON \
  -DKokkos_ARCH_SKX=ON \
  <KOKKOS_SOURCE>
> make install

Then build Kokkos Kernels, pointing to Kokkos:

> cmake \
  -DKokkos_ROOT=${HOME}/kokkos-install \
  -DCMAKE_CXX_COMPILER=g++ 
  <KOKKOS_KERNELS_SOURCE>
> make

Cuda and Serial backends, nvcc_wrapper compiler, Power8 and Volta sm_70 architectures, various compilation flags

First install Kokkos:

> cmake \
  -DCMAKE_CXX_COMPILER=<KOKKOS_SOURCE>/bin/nvcc_wrapper \
  -DCMAKE_INSTALL_PREFIX=${HOME}/kokkos-install \
  -DKokkos_ENABLE_CUDA=ON \
  -DKokkos_ENABLE_SERIAL=ON \
  -DKokkos_ARCH_VOLTA70=ON \
  -DKokkos_ARCH_POWER8=ON \
  <KOKKOS_SOURCE>
> make install

Then build Kokkos Kernels, pointing to Kokkos:

> cmake \
  -DKokkos_ROOT=${HOME}/kokkos-install \
  -DCMAKE_CXX_COMPILER=${HOME}/kokkos-install/bin/nvcc_wrapper \
  <KOKKOS_KERNELS_SOURCE>
> make

If you wish to enable certain CUDA third-party libraries (TPLs), you can also configure with

> cmake \
  -DKokkos_ROOT=${HOME}/kokkos-install \
  -DCMAKE_CXX_COMPILER=${HOME}/kokkos-install/bin/nvcc_wrapper \
  -DKokkosKernels_ENABLE_TPL_CUSPARSE=ON \
  <KOKKOS_KERNELS_SOURCE>

Required

  • CMake >= 3.10
  • BLAS, LAPACK (usually)
  • Compatible compiler and hardware
  • CUDA support currently requires UVM for most Trilinos packages (Kokkos and KokkosKernels do not have this requirement)

Trilinos

If building with Trilinos, the same set of CMake options apply. The only difference is you must enable KokkosKernels:

  cmake \
    -D Trilinos_ENABLE_KokkosKernels:BOOL=ON \
  ...

Running tests:

Note, no tests will be available unless -DKokkosKernels_ENABLE_TESTS=ON is in your cmake command. To run the tests, simply execute a CMake build and then run:

> make test

To limit the tests, one can cd into either unit_test or perf_test and also run make test. To show full detail of all tests, you can run ctest --extra-verbose. You can filter exactly which tests are run based on regular expressions with

ctest -R <match_string>

Tests are grouped into individual executables. You can run the executable for one of the enabled backends based on your configuration, for example if OpenMP is enabled:

>./KokkosKernels_UnitTest_OpenMP

To run a specific test in the executable use the --gtest_filter flag:

>./KokkosKernels_UnitTest_OpenMP --gtest_filter=openmp.dot_double`

Kokkos Kernels CMake Option Listing

  • BLAS_LIBRARIES: STRING
    • Optional override for the libraries that comprise TPL BLAS.
    • Default: None. Default common library names will be searched
  • BLAS_LIBRARY_DIRS: STRING
    • Optional override for the library directories that comprise TPL BLAS.
    • Default: None. Default common library locations will be searched
  • CUBLAS_LIBRARIES: STRING
    • Optional override for the libraries that comprise TPL CUBLAS.
    • Default: None. Default common library names will be searched
  • CUBLAS_LIBRARY_DIRS: STRING
    • Optional override for the library directories that comprise TPL CUBLAS.
    • Default: None. Default common library locations will be searched
  • CUSPARSE_LIBRARIES: STRING
    • Optional override for the libraries that comprise TPL CUSPARSE.
    • Default: None. Default common library names will be searched
  • CUSPARSE_LIBRARY_DIRS: STRING
    • Optional override for the library directories that comprise TPL CUSPARSE.
    • Default: None. Default common library locations will be searched
  • KokkosKernels_BLAS_ROOT: PATH
    • Location of BLAS install root.
    • Default: None or the value of the environment variable BLAS_ROOT if set
  • KokkosKernels_CUBLAS_ROOT: PATH
    • Location of CUBLAS install root.
    • Default: None or the value of the environment variable CUBLAS_ROOT if set
  • KokkosKernels_CUSPARSE_ROOT: PATH
    • Location of CUSPARSE install root.
    • Default: None or the value of the environment variable CUSPARSE_ROOT if set
  • KokkosKernels_ENABLE_EXAMPLES: BOOL
    • Whether to build examples.
    • Default: OFF
  • KokkosKernels_ENABLE_EXPERIMENTAL: BOOL
    • Enable building and installation of experimental KokkosKernels features.
    • Default: OFF
  • KokkosKernels_ENABLE_TESTS: BOOL
    • Whether to build tests.
    • Default: OFF
  • KokkosKernels_ENABLE_TPL_BLAS: BOOL
    • Whether to enable BLAS
    • Default: OFF
  • KokkosKernels_ENABLE_TPL_CUBLAS: BOOL
    • Whether to enable CUBLAS
    • Default: ON if CUDA-enabled Kokkos, otherwise OFF
  • KokkosKernels_ENABLE_TPL_CUSPARSE: BOOL
    • Whether to enable CUSPARSE
    • Default: ON if CUDA-enabled Kokkos, otherwise OFF
  • KokkosKernels_ENABLE_TPL_LAPACK: BOOL
    • Whether to enable LAPACK
    • Default: ON if BLAS is enabled, otherwise OFF
  • KokkosKernels_ENABLE_TPL_MAGMA: BOOL
    • Whether to enable MAGMA
    • Default: OFF
  • KokkosKernels_ENABLE_TPL_MKL: BOOL
    • Whether to enable MKL
    • Default: OFF
  • KokkosKernels_ETI_ONLY: BOOL
    • Whether to restrict availability of kernels to ETI types only. Turning this on guarantees that kernels are never built inside of object files which simply call KokkosKernels functions.
    • Default: OFF
  • KokkosKernels_INST_COMPLEX_DOUBLE: BOOL
    • Whether to pre instantiate kernels for the scalar type complex. Disabling this may increase build times.
    • Default: OFF or unless enabled during a Trilinos build with Trilinos_ENABLE_COMPLEX_DOUBLE.
  • KokkosKernels_INST_COMPLEX_FLOAT: BOOL
    • Whether to pre instantiate kernels for the scalar type complex. Disabling this may increase build times.
    • Default: OFF or unless enabled during a Trilinos build with Trilinos_ENABLE_COMPLEX_FLOAT.
  • KokkosKernels_INST_DOUBLE: BOOL
    • Whether to pre instantiate kernels for the scalar type double. This option is KokkosKernels_INST_DOUBLE=ON by default. Disabling this may increase build times.
    • Default: ON
  • KokkosKernels_INST_EXECSPACE_OPENMP: BOOL
    • Whether to pre instantiate kernels for the execution space Kokkos::OpenMP. Disabling this when Kokkos_ENABLE_OpenMP is enabled may increase build times.
    • Default: ON if Kokkos is OpenMP-enabled, OFF otherwise.
  • KokkosKernels_INST_EXECSPACE_SERIAL: BOOL
    • Whether to build kernels for the execution space Kokkos::Serial. If explicit template instantiation (ETI) is enabled in Trilinos, disabling this when Kokkos_ENABLE_SERIAL is enabled may increase build times.
    • Default: ON when Kokkos is Serial-enabled, OFF otherwise.
  • KokkosKernels_INST_EXECSPACE_THREADS: BOOL
    • Whether to build kernels for the execution space Kokkos::Threads. If explicit template instantiation (ETI) is enabled in Trilinos, disabling this when Kokkos_ENABLE_PTHREAD is enabled may increase build times.
    • Default: ON if Kokkos is Threads-enabled, OFF otherwise.
  • KokkosKernels_INST_FLOAT: BOOL
    • Whether to pre instantiate kernels for the scalar type float. Disabling this may increase build times.
    • Default: OFF or unless enabled during a Trilinos build with Trilinos_ENABLE_FLOAT.
  • KokkosKernels_INST_LAYOUTLEFT: BOOL
    • Whether to pre instantiate kernels for the view layout LayoutLeft. This option is KokkosKernels_INST_LAYOUTLEFT=ON by default. Disabling this may increase build times.
    • Default: ON
  • KokkosKernels_INST_LAYOUTRIGHT: BOOL
    • Whether to pre instantiate kernels for the view layout LayoutRight. This option is KokkosKernels_INST_LAYOUTRIGHT=OFF by default. Disabling this may increase build times.
    • Default: OFF
  • KokkosKernels_INST_MEMSPACE_HOSTSPACE: BOOL
    • Whether to pre instantiate kernels for the memory space Kokkos::HostSpace. Disabling this when one of the Host execution spaces is enabled may increase build times.
    • Default: ON
  • KokkosKernels_INST_OFFSET_INT: BOOL
    • Whether to pre instantiate kernels for the offset type int. This option is KokkosKernels_INST_OFFSET_INT=ON by default.
    • Default: ON
  • KokkosKernels_INST_OFFSET_SIZE_T: BOOL
    • Whether to pre instantiate kernels for the offset type size_t. This option is KokkosKernels_INST_OFFSET_SIZE_T=OFF by default.
    • Default: ON
  • KokkosKernels_INST_ORDINAL_INT: BOOL
    • Whether to pre instantiate kernels for the ordinal type int. This option is KokkosKernels_INST_ORDINAL_INT=ON by default.
    • Default: ON
  • KokkosKernels_INST_ORDINAL_INT64_T: BOOL
    • Whether to pre instantiate kernels for the ordinal type int64_t. This option is KokkosKernels_INST_ORDINAL_INT64_T=OFF by default.
    • Default: OFF
  • KokkosKernels_LAPACK_ROOT: PATH
    • Location of LAPACK install root.
    • Default: None or the value of the environment variable LAPACK_ROOT if set
  • KokkosKernels_LINALG_OPT_LEVEL: BOOL
    • Optimization level for KokkosKernels computational kernels: a nonnegative integer. Higher levels result in better performance that is more uniform for corner cases, but increase build time and library size. The default value is 1, which should give performance within ten percent of optimal on most platforms, for most problems.
    • Default: 1
  • KokkosKernels_MAGMA_ROOT: PATH
    • Location of MAGMA install root.
    • Default: None or the value of the environment variable MAGMA_ROOT if set
  • KokkosKernels_MKL_ROOT: PATH
    • Location of MKL install root.
    • Default: None or the value of the environment variable MKL_ROOT if set
  • KokkosKernels_NO_DEFAULT_CUDA_TPLS: BOOL
    • Whether CUDA TPLs should be enabled by default.
    • Default: OFF
  • KokkosKernels_TEST_ETI_ONLY: BOOL
    • Whether to restrict testing to ETI types.
    • Default: ON
  • LAPACK_LIBRARIES: STRING
    • Optional override for the libraries that comprise TPL LAPACK.
    • Default: None. Default common library names will be searched
  • LAPACK_LIBRARY_DIRS: STRING
    • Optional override for the library directories that comprise TPL LAPACK.
    • Default: None. Default common library locations will be searched
  • MKL_LIBRARIES: STRING
    • Optional override for the libraries that comprise TPL MKL.
    • Default: None. Default common library names will be searched
  • MKL_LIBRARY_DIRS: STRING
    • Optional override for the library directories that comprise TPL MKL.
    • Default: None. Default common library locations will be searched

Supported Third Party Libraries:

TPL Name Min Version Max Version
TPL1 i.j.k I.J.K
TPL2 x.y.z X.Y.Z

Home:

Using KokkosKernels

API Reference

Examples

Benchmarks

Clone this wiki locally