Adding New Backends to Spatter

Last updated: September 6th, 2024

Spatter is a relatively simple benchmark suite, at least from a kernel standpoint. Scatter and Gather are the main benchmark kernels that are used, but there are additional helper files that are needed to add new accelerator or device backends. Here we briefly review the steps to add a new "accelerator" backend called <newgpu> starting from the CUDA-based GPU backend.

NOTE: We are happy to help you work with a "draft" PR, so please feel free to open one for further discussion.

Adding a new backend:

1. Fork the Spatter repo and create a new feature branch on your local repo git clone <myspatterfork.git> && git checkout -b newgpu
2. Add new code for the backend
3. Update CMake and compiler options to compile your backend. Test locally.
4. Write and add unit tests for your backend. Use CRNCH RG testbed resources, as needed to test
5. Update documentation, submit your PR, and work with maintainers to get it merged.

Step 2 - Code additions

In some cases, you can use translation tools like SYCLOMatic or hipify to convert CUDA code to a new backend - these files are denoted with a bold note. However, many of the listed files here are configuration and initialization files that just require copying the CUDA version of the code and updating it for your new backend. Look for USE_CUDA flags throughout these files for examples of what you need to modify/replicate.

The most important backend files are as follows:

• src/main.cc - contains some basic backend output and GPU-specific initialization. Modify initialization/debug information for your backend.
• src/Spatter/Configuration.hh - modify this file to create your own ConfigurationBase object for your <newgpu> backend.
• src/Spatter/Configuration.cc - look for the USE_CUDA flag and copy/modify these functions for your <newgpu> backend. Note that you will need to replicate about 200 lines of code to prototype all the initialization, gather, scatter, and error checking functions. Candidate for using a translation tool
• src/Spatter/CudaBackend.cu - copy this file and create your own version for your <newgpu> backend. Candidate for translation tool.
• src/Spatter/CudaBackend.hh - contains prototype for backend-specific kernels but uses standard C++ code only. copy this file and create your own version for your <newgpu> backend.
• src/Spatter/Input.hh - contains options for selecting different backends and creating their Configuration objects. Modify this file for your <newgpu> backend.
• src/Spatter/JsonParser.hh - similar to Input.hh, this file contains JSON-specific options for selecting different backends and creating their Configuration objects. Modify this file for your <newgpu> backend.
• src/Spatter/SpatterTypes.hh - add a specific struct object for your <newgpu> backend.

Step 3 - Compiler and CMake Additions

Important CMake files you need to modify include:

• CMakeLists.txt - top-level CMakeLists file that points to CUDA pkgs with include(pkgs/CUDASupport). Add your own backend pkg include
• cmake/pkgs/CUDASupport.cmake - contains CMake calls to find a local CUDA installation and sets the USE_CUDA flags. Add your own backend version of this file and point to it from the top-level CMakeLists.txt
• cmake/CompilerType.cmake - includes specific optimization flags for different compilers. Update if your compiler uses special flags to optimize gather/scatter or if -O3 is not desired as the default compilation option.
• src/Spatter/CMakeLists.txt - contains files to add CUDA backend files and to link them. Look for USE_CUDA regions and replicate for your backend
• tests/CMakeLists.txt - adds build options for backend-specific tests. Look for the example with USE_CUDA and add your own USE_NEWGPU ifdef section pointing to your new tests (detailed below in Section 4).

Step 4 - Unit Test Additions

You need to add a few unit tests for your backend so that it can be tested using our CI/CD setup. These files are important examples you should copy or modify:

• tests/parse_run_config_suite_gpu.cc - copy this file and create your own file tests/parse_run_config_suite_<newgpu>.cc. You may want to add/modify the specific configuration options to match your Configuration setup in src/Spatter/Configuration.cc
• tests/standard_suite_gpu.cc - copy this file and create your own file tests/standard_suite_<newgpu>.cc. You must have at least a working stream and ustride test for us to merge your PR. If you are able to run the other patterns as well that is highly preferred.
• (Optional) tests/misc/run-cuda-crnch.sh - if you are using CRNCH RG testbed to test your code on a specific GPU, check out the instructions for setting up a self-hosted runner. We can help you do this with a draft PR as well.

Step 5- Documentation and PR

1. Document options for your new backend in Build.md and in the README.

2. Add yourself to the AUTHORS list in your PR with a brief mention of what you've contributed.

3. Commit your new code to your branch and push to your local repository.

4. Test the build locally including running the tests. Confirm that these complete correctly.

5. Create a pull request for the main repo. Make requested changes from review until your code can be safely merged into the main branch.