undefined reference to `cuDevicePrimaryCtxRelease_v2'

[singlebook]

Hello all,
I try to compile cp2k with CUDA Support. My compiler is gcc 9.3.0 with cuda 11.1.

I always get the error:

**/cp2k-8.1/lib/local_cuda/psmp/exts/dbcsr/libdbcsr.a(acc_init.o): In function `acc_finalize':

acc_init.cpp:(.text+0xf6): undefined reference to `cuDevicePrimaryCtxRelease_v2'

collect2: error: ld returned 1 exit status**

How could I solve this problem?

Thank you very much!

[alazzaro]

Could you elaborate a bit more?
For instance, how you compile CP2K (please note that this is the DBCSR forum. You better move your request to the CP2K forum)

[alazzaro]

A bit of more investigation, it seems that you are missing cuBLAS library somehow.
I found some blogs that are reporting similar problems. The route cause is that with CUDA 11 now cuBLAS is on a different path (see https://docs.nvidia.com/cuda/archive/11.0/cuda-toolkit-release-notes/index.html#cublas-new-features). Could you check that?

[singlebook]

I compared the differences between cuda-10.0 and cuda-11.1. Both are installed properly.

There are some new macrodefinitions in cuda.h of cuda-11.1, such as
#define cuDevicePrimaryCtxRelease cuDevicePrimaryCtxRelease_v2

There is no such thing in that of cuda-10.0. So when I try to compile dbcsr in cp2k-8.1 with cuda support , it gives me that error.

[alazzaro]

This is good to know but it brings more questions and considerations:

1. do you get the error with CUDA 10? (I assume no)
2. Do you have cUDA 10 and CUDA 11 installed on the same system?
3. the _v2 is fine, it was introduced long back ago. For instance, inside the cuda.h of CUDA 10 there are already a lot of defines with _v2. I assume CUDA introduced this new function cuDevicePrimaryCtxRelease_v2 with CUDA 11.1, so we need to know where this function is defined.
4. I found a CUDA 11 installation, from what I can see now the library is installed under the standard directory /usr/lib64, where I have libcuda.so.450.80.02. If I check for the symbols there, I have:

> readelf -Ws libcuda.so.450.80.02 | grep cuDevicePrimaryCtxRelease_v2
   301: 00000000002b55f0   425 FUNC    GLOBAL DEFAULT   11 cuDevicePrimaryCtxRelease_v2

So it is declared. Make sure you have this library with the correct symbol in there, and it is available in the path when linking.

Now, as far I can see, this is not a DBCSR problem itself, neither a CP2K problem. It looks like a problem on the CUDA side.
So I suggest posting your request on the CP2K google forum (https://groups.google.com/g/cp2k), where maybe someone who is using CUDA 11 can give you some advice. I'm closing the issue here.

[dev-zero]

maybe as a help: we tested CP2K with CUDA-11.1 not too long ago and it worked.

The arch file as generated by the toolchain was:

CC          = gcc
CXX         = g++
AR          = ar -r
FC          = gfortran
LD          = gfortran
#
DFLAGS      = -D__GRID_CUDA -D__ACC -D__DBCSR_ACC -D__PW_CUDA  -D__LIBXSMM   -D__FFTW3  -D__LIBINT -D__LIBXC   
#
WFLAGS      = -Werror=aliasing -Werror=ampersand -Werror=c-binding-type -Werror=intrinsic-shadow -Werror=intrinsics-std -Werror=line-truncation -Werror=tabs -Werror=target-lifetime -Werror=underflow -Werror=unused-but-set-variable -Werror=unused-variable -Werror=unused-dummy-argument -Werror=conversion -Werror=zerotrip -Wno-maybe-uninitialized -Wuninitialized -Wuse-without-only 
#
FCDEBFLAGS  = -fbacktrace -ffree-form -fimplicit-none -std=f2008 
CFLAGS      = -march=native -mtune=native -fno-omit-frame-pointer -g  -O3 -funroll-loops $(PROFOPT) -fopenmp   -I'/data/localuser/cp2k/tools/toolchain/install/openblas-0.3.10/include' -I'/data/localuser/cp2k/tools/toolchain/install/fftw-3.3.8/include' -I'/data/localuser/cp2k/tools/toolchain/install/libint-v2.6.0-cp2k-lmax-5/include' -I'/data/localuser/cp2k/tools/toolchain/install/libxc-4.3.4/include' -I'/data/localuser/cp2k/tools/toolchain/install/libxsmm-1.16.1/include' -std=c99 -Wall -Wextra -Werror $(DFLAGS) 
FCFLAGS     = -march=native -mtune=native -fno-omit-frame-pointer -g  -O3 -funroll-loops $(PROFOPT) -fopenmp   -I'/data/localuser/cp2k/tools/toolchain/install/openblas-0.3.10/include' -I'/data/localuser/cp2k/tools/toolchain/install/fftw-3.3.8/include' -I'/data/localuser/cp2k/tools/toolchain/install/libint-v2.6.0-cp2k-lmax-5/include' -I'/data/localuser/cp2k/tools/toolchain/install/libxc-4.3.4/include' -I'/data/localuser/cp2k/tools/toolchain/install/libxsmm-1.16.1/include' $(FCDEBFLAGS) $(WFLAGS) $(DFLAGS)
#
LDFLAGS     = $(FCFLAGS) -Wl,--enable-new-dtags   -L'/data/localuser/cp2k/tools/toolchain/install/openblas-0.3.10/lib' -Wl,-rpath='/data/localuser/cp2k/tools/toolchain/install/openblas-0.3.10/lib' -L'/data/localuser/cp2k/tools/toolchain/install/fftw-3.3.8/lib' -Wl,-rpath='/data/localuser/cp2k/tools/toolchain/install/fftw-3.3.8/lib' -L'/data/localuser/cp2k/tools/toolchain/install/libint-v2.6.0-cp2k-lmax-5/lib' -L'/data/localuser/cp2k/tools/toolchain/install/libxc-4.3.4/lib' -Wl,-rpath='/data/localuser/cp2k/tools/toolchain/install/libxc-4.3.4/lib' -L'/data/localuser/cp2k/tools/toolchain/install/libxsmm-1.16.1/lib' -Wl,-rpath='/data/localuser/cp2k/tools/toolchain/install/libxsmm-1.16.1/lib'  -L'/usr/local/cuda-11.1/targets/x86_64-linux/lib' -Wl,-rpath='/usr/local/cuda-11.1/targets/x86_64-linux/lib' -L'/usr/lib64' -Wl,-rpath='/usr/lib64'
LIBS        =  -lxsmmf -lxsmm -ldl -lpthread -lxcf03 -lxc -lint2  -lfftw3 -lfftw3_omp    -lopenblas -lstdc++ -lcudart -lnvrtc -lcuda -lcufft -lcublas -lrt 
#
CXX         = gcc
CXXFLAGS    = -O2 -fno-omit-frame-pointer -g -march=native -mtune=native -I${CUDA_PATH}/include -std=c++11 -fopenmp
CFLAGS      = -march=native -mtune=native -fno-omit-frame-pointer -g  -O3 -funroll-loops $(PROFOPT) -fopenmp   -I'/data/localuser/cp2k/tools/toolchain/install/openblas-0.3.10/include' -I'/data/localuser/cp2k/tools/toolchain/install/fftw-3.3.8/include' -I'/data/localuser/cp2k/tools/toolchain/install/libint-v2.6.0-cp2k-lmax-5/include' -I'/data/localuser/cp2k/tools/toolchain/install/libxc-4.3.4/include' -I'/data/localuser/cp2k/tools/toolchain/install/libxsmm-1.16.1/include' -std=c99 -Wall -Wextra -Werror $(DFLAGS)  -I${CUDA_PATH}/include
GPUVER      = K20X
NVCC        = nvcc
NVFLAGS     = -arch sm_35 -O3 -Xcompiler='-fopenmp' --std=c++11 $(DFLAGS)
#
FYPPFLAGS   = -n --line-marker-format=gfortran5

Maybe compare the LIBS line. Also, when reporting to https://groups.google.com/group/cp2k please also include your arch file and the complete build.

[hfp]

Things like this can also happen if one compiles on a "head node" with different/no CUDA runtime/driver but using the latest CUDA headers.