Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nightly test failure, Cuda builds with Cu* TPLs enabled, Cuda.sparse_sptrsv_double_int_int_TestDevice #2442

Closed
ndellingwood opened this issue Nov 26, 2024 · 0 comments · Fixed by #2444

Comments

@ndellingwood
Copy link
Contributor

Nightly Cuda builds with TPLs enabled (various versions and arch) are failing the sparse_cuda test in the sptrsv check:

cuda/11.2.2 w/ tpls cusparse, cublas, cusolver on Volta70:

07:23:30 [ RUN      ] Cuda.sparse_sptrsv_double_int_int_TestDevice
07:23:30 /home/jenkins/weaver/workspace/KokkosKernels_Weaver_Cuda_Serial_cuda_1122_gcc_831_cusparse_cublas_cusolver_all_scalars/kokkos-kernels/sparse/unit_test/Test_Sparse_sptrsv.hpp:604: Failure
07:23:30 Expected equality of these values:
07:23:30   sum
07:23:30     Which is: -nan
07:23:30   lhs_v[i].extent(0)
07:23:30     Which is: 6
07:23:30 /home/jenkins/weaver/workspace/KokkosKernels_Weaver_Cuda_Serial_cuda_1122_gcc_831_cusparse_cublas_cusolver_all_scalars/kokkos-kernels/sparse/unit_test/Test_Sparse_sptrsv.hpp:604: Failure
07:23:30 Expected equality of these values:
07:23:30   sum
07:23:30     Which is: -nan
07:23:30   lhs_v[i].extent(0)
07:23:30     Which is: 6

Also seen e.g. with cuda/12.2 on A100 (Ampere)

This follows after merge of #2429 , @jgfouca can you investigate?

Reproducer: (weaver rhel8 queue):

bsub -Is -n 1 -q rhel8 -gpu "num=1" bash

source /etc/profile.d/modules.sh
source /projects/ppc64le-pwr9-rhel8/legacy-env.sh
module load cuda/11.2.2/gcc/8.3.1 cmake/3.23.1

$KOKKOSKERNELS_PATH/cm_generate_makefile.bash --with-cuda --with-serial --compiler=$KOKKOS_PATH/bin/nvcc_wrapper --arch=Volta70,Power9 --with-cuda-options=enable_lambda --kokkos-path=$KOKKOS_PATH --kokkoskernels-path=$KOKKOSKERNELS_PATH  --with-scalars='double,complex_double,float,complex_float' --with-ordinals=int --with-offsets=int,size_t --with-layouts=LayoutLeft --with-tpls=cusparse,cublas,cusolver --cxxstandard=17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants