LuisaRenderPy CUDA backend won't install

[kaspar-gestalt]

I followed the instructions

I have an Ubuntu20.04.

NVIDIA-SMI 535.183.01 Driver Version: 535.183.01 CUDA Version: 12.2

NVIDIA GeForce GTX 1080 Ti

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

[wangyian-me]

Try coda 12.1 instead of 10.1 perhaps?
Also, I met with other problems before running Luisa on 1080 Ti even with correct cuda and environment.

[oscar-lima]

I also have Ubuntu 20.04 and I got LuisaRenderPy to acknowledge my installation of CUDA, so I thought I share at least that part:

I followed steps described in https://genesis-world.readthedocs.io/en/latest/user_guide/getting_started/visualization.html#photo-realistic-ray-tracing-rendering

up until:

Build LuisaRender. Remember to use the correct cmake.

meaning, do NOT try:

cmake -S . -B build -D CMAKE_BUILD_TYPE=Release -D PYTHON_VERSIONS=3.9 -D LUISA_COMPUTE_DOWNLOAD_NVCOMP=ON # remember to check python version

because you want to carefully configure CUDA first. If you do, the following warnings appear:

LuisaRenderPy cmake CUDA warnings

During cmake configuration of LuisaRenderPy the following warning appeared:

foo@bar $ cmake -S . -B build -D CMAKE_BUILD_TYPE=Release -D PYTHON_VERSIONS=3.10 -D LUISA_COMPUTE_DOWNLOAD_NVCOMP=ON -D LUISA_COMPUTE_ENABLE_GUI=OFF -DCMAKE_INSTALL_PREFIX=/home/oscar/.pyenv/versions/genesis-sim/lib/python3.10/site-packages
.
.
CMake Warning at src/compute/scripts/validate_options.cmake:70 (message):
  The CUDA backend is not available.  The CUDA backend will be disabled.
Call Stack (most recent call first):
  src/compute/scripts/validate_options.cmake:98 (report_feature_not_available)
  src/compute/CMakeLists.txt:53 (include)
.
.
.

However LuisaRenderPy compiles despite the warnings above, at this stage if you try the rendering demo.py you get a Runtime error and the program crashes:

foo@bar $ python demo.py --vis
.
.
[2024-12-24 08:32:00.444] [console] [error] Backend 'cuda' is not installed. [... Genesis/genesis/ext/LuisaRender/src/compute/src/runtime/context.cpp:77]
.
.
.
Aborted (core dumped)

So LuisaRenderPy really needs a clean, solid CUDA setup to work. Based on @wangyian-me's comment, I gave CUDA 12.1 a try. However, from #207, I see that people are trying CUDA 12.2 with success (for Ubuntu 22.04).

Check the maximum CUDA version supported by the installed NVIDIA driver

One thing one needs to be careful about is that when doing nvidia-smi the CUDA version that is printed is the maximum CUDA version supported by the installed NVIDIA driver and NOT your "active" CUDA version, e.g.,:

foo@bar $ nvidia-smi
Tue Dec 24 00:05:39 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2(A)  |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Quadro RTX 4000 with Max...    Off | 00000000:01:00.0  On |                  N/A |
| N/A   48C    P8               8W /  30W |    853MiB /  8192MiB |      3%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      3738      G   /usr/lib/xorg/Xorg                          132MiB |
|    0   N/A  N/A      5276      G   /usr/lib/xorg/Xorg                          346MiB |
|    0   N/A  N/A      5408      G   /usr/bin/gnome-shell                        129MiB |
|    0   N/A  N/A      9525      G   /usr/lib/firefox/firefox                    167MiB |
|    0   N/A  N/A    628369      G   ...erProcess --variations-seed-version       68MiB |
+---------------------------------------------------------------------------------------+

(A) : the maximum CUDA version supported by the installed NVIDIA driver, i.e.:
      Driver Version 535.183.01 corresponds to a driver that supports up to CUDA 12.2

How to check your "active" cuda version?

do nvcc --version:

foo@bar $ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

Answer: 10.1

How to install CUDA 12.1 (for Ubuntu 20.04)?

from: https://developer.nvidia.com/cuda-12-1-0-download-archive?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=20.04&target_type=deb_local

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/12.1.0/local_installers/cuda-repo-ubuntu2004-12-1-local_12.1.0-530.30.02-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2004-12-1-local_12.1.0-530.30.02-1_amd64.deb
sudo cp /var/cuda-repo-ubuntu2004-12-1-local/cuda-D8BCD751-keyring.gpg /usr/share/keyrings/ # this was suggested by previous step
sudo cp /var/cuda-repo-ubuntu2004-12-1-local/cuda-*-keyring.gpg /usr/share/keyrings # this was in the CUDA wiki
sudo apt-get update
sudo apt-get -y install cuda-toolkit-12-1 # NOTE (A)
rm cuda-repo-ubuntu2004-12-1-local_12.1.0-530.30.02-1_amd64.deb # optional cleanup

NOTE (A): official instructions say: sudo apt-get -y install cuda # but this might give error:

The following packages have unmet dependencies:
 cuda : Depends: cuda-12-3 (>= 12.3.2) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

and this is why one has to do sudo apt-get -y install cuda-toolkit-12-1 instead.

Check which cuda toolkits are installed in your system

foo@bar $ ls /usr/local | grep cuda
cuda
cuda-12
cuda-12.1
cuda-12.3

For this example, cuda 12, 12.1 and 12.3 are installed on the system, note that cuda folder is a softlink that points to the globally active installed CUDA. In other words to set a default CUDA version on your system do:

sudo ln -s /usr/local/cuda-12.1 /usr/local/cuda

Which will create a softlink /usr/local/cuda that points to /usr/local/cuda-12.1.
Note that obviously this is a persistent change and sets a "global" installation of CUDA on your system.

Warning: if for some reason you ran sudo apt install nvidia-cuda-toolkit (probably sometime in the distant past like me ...), then you get an old CUDA version (10.1 in my case) which lives under /usr/bin/nvcc and points to /usr/lib/nvidia-cuda-toolkit/bin/nvcc. You most definitely don't want that for LuisaRenderPy.

To override, you can either remove nvidia-cuda-toolkit, (be careful and make sure other software installed on your system is not using it), or you can set your environment to set a preference of /usr/local/cuda/bin over /usr/bin by doing:

export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

NOTE: the order of paths within PATH env matters.

Make sure nvcc points to CUDA 12.1

After you configured correctly CUDA/env (using the commands described earlier), make sure that nvcc points to CUDA 12.1:

foo@bar $ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Feb__7_19:32:13_PST_2023
Cuda compilation tools, release 12.1, V12.1.66
Build cuda_12.1.r12.1/compiler.32415258_0

As you can see, CUDA 12.1 is now "active".

Configure and build LuisaRenderPy

This is were you want to try the rest of the steps described in https://genesis-world.readthedocs.io/en/latest/user_guide/getting_started/visualization.html#photo-realistic-ray-tracing-rendering

Starting from:

Build LuisaRender. Remember to use the correct cmake.

Meaning, do try:

cmake -S . -B build -D CMAKE_BUILD_TYPE=Release -D PYTHON_VERSIONS=3.9 -D LUISA_COMPUTE_DOWNLOAD_NVCOMP=ON # remember to check python version
cmake --build build -j $(nproc)

but added -D CMAKE_CUDA_COMPILER=/usr/local/cuda-12.1/bin/nvcc and -D LUISA_COMPUTE_ENABLE_CUDA=ON

Since I used pyenv, I used slighlty different commands:

cd Genesis/genesis/ext/LuisaRender
mkdir build
cd build
cmake -D CMAKE_INSTALL_PREFIX=/home/oscar/.pyenv/versions/genesis-sim/lib/python3.10/site-packages -D CMAKE_BUILD_TYPE=Release -D LUISA_COMPUTE_ENABLE_CUDA=ON -D PYTHON_VERSIONS=3.10 -D Python_EXECUTABLE=/home/oscar/.pyenv/versions/genesis-sim/bin/python -D LUISA_COMPUTE_DOWNLOAD_NVCOMP=ON -D CMAKE_CUDA_COMPILER=/usr/local/cuda-12.1/bin/nvcc .. # configure
cmake --build . -j$(nproc) # build

Finally CUDA was found!

-- Found CUDAToolkit: /usr/local/cuda-12.1/targets/x86_64-linux/include (found suitable version "12.1.66", minimum required is "11.7")
-- Build with CUDA backend: 12.1.66
-- Downloading nvcomp from https://developer.download.nvidia.com/compute/nvcomp/3.0.4/local_installers/nvcomp_3.0.4_x86_64_12.x.tgz
-- Populating nvcomp

Now, to be honest... I still have vulkan and other issues (see F), however since this issue is concerned with "LuisaRenderPy CUDA backend won't install" (for ubuntu 20.04), I think I covered at least that part...

F:

Unresolved Warnings:

- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed

- The Vulkan backend is not available.  The Vulkan backend will be disabled.
I partially solved it? : the warning is gone but now it does not build... so far I did:
git clone https://github.com/KhronosGroup/Vulkan-Headers.git
cd Vulkan-Headers
mkdir -p /usr/include/vulkan
cp include/vulkan/* /usr/include/vulkan

- Could not find sccache

Update: I resolved some warnings:

No package 'zzip-zlib-config' found
Solved via: sudo apt install libzzip-dev

The DirectX backend is not available.  The DirectX backend will be disabled.
Solved: I am a Linux user, DirecX is for Windows only, you can suppress the warning with the following cmake flag:
-D LUISA_COMPUTE_ENABLE_DX=OFF

The Metal backend is not available.  The Metal backend will be disabled.
Solved: I am a Linux user, Metal is only for Mac, suppress warning by adding a cmake flag:
-D LUISA_COMPUTE_ENABLE_METAL=OFF

NVTT not found! Please install NVTT from
  https://developer.nvidia.com/nvidia-texture-tools-exporter and set the
  CMake NVTT_DIR variable to the folder containing nvtt*.dll (e.g.
  C:\Program Files\NVIDIA Corporation\NVIDIA Texture Tools) .
Solved:
As suggested by the warning I went to https://developer.nvidia.com/nvidia-texture-tools-exporter
downloaded Nvidia texture tools (NVIDIA_Texture_Tools_Linux_x86_64_3.2.5)

and

foo@bar $ locate libnvtt.so
/home/user/Downloads/NVIDIA_Texture_Tools_Linux_x86_64_3.2.5/libnvtt.so.30205

now the question is: how to tell cmake about it?

from: /home/user/.../Genesis/genesis/ext/LuisaRender/src/compute/src/backends/cuda/CMakeLists.txt line 133:

set(_SEARCH_PATHS
    "${CMAKE_CURRENT_LIST_DIR}"
    "${CMAKE_CURRENT_LIST_DIR}/.."
    "${CMAKE_CURRENT_LIST_DIR}/../nvtt"
    "C:/Program Files/NVIDIA Corporation/NVIDIA Texture Tools"
    "$ENV{NVTT_DIR}")

apparently there are many options, place NVIDIA_Texture_Tools_Linux_x86_64_3.2.5 folder in:
- the same directory
- one back
- one back + /nvtt
- some windows fishy stuff (hurray for linux!)
- set an environment variable named NVTT_DIR that points to NVIDIA_Texture_Tools_Linux_x86_64_3.2.5
export NVTT_DIR=/home/oscar/software/NVIDIA_Texture_Tools_Linux_x86_64_3.2.5
it worked! now nvtt is found and warning is gone.

[kaspar-gestalt]

Cool, thanks @oscar-lima @wangyian-me .

I installed CUDA toolkit 12.6 and CUDA drivers 565.57.01, and re-built LuisaRenderPy. Now I am encountering the error [error] CUDA_ERROR_INVALID_SOURCE: device kernel image is invalid [/home/user/repos/Genesis/genesis/ext/LuisaRender/src/compute/src/backends/cuda /cuda_shader_native.cpp:46], when I try to cam.render(...)

Is that directly related, or should I open a new issue?

[kaspar-gestalt]

@oscar-lima Does the actual compilation of LuisaRender work now with NVTT enabled, or just its build configuration? I'm asking because my configuration also seems find, with NVTT being found, but the compilation errors out: #465