Nvidia passthrough & cuda working, but nvtop says "No GPU to monitor"

[Kailee71]

As title.
cuda is working perfectly and I'm enjoying doing some compute tasks on this machine now, but would like to monitor using nvtop, which says "no gpu to monitor". Nvidia-smi works flawlessly. Installing nvidia-drivers-* fails in the jail, so that's no help either...
Anybody have an idea why nvtop won't work even though nvidia-smi works?

Kai.

[Jip-Hop]

Sorry I have no idea as I have no nvidia GPU. But on a related note: Could you please try the jlmkr.py file from the develop branch? I think it should now load the nvidia kernel module automatically. Which means you can disable the Post Init script to load the kernel module at boot.

[Kailee71]

Oooh cool yes - I'll try it right now :-)

This doesn't look good. I just downloaded it with wget and chmodded it - is there anything else I need to do?

[Kailee71]

Does this console message maybe have something to do with nvtop not wotking?

btw the way I went back to the standard jlmkr.py, rebootet with the post-init still disabled, and "it just works" anyway.

[Jip-Hop]

Those logs don't look good. This is what it looks like to me:

curl --location --remote-name https://raw.githubusercontent.com/Jip-Hop/jailmaker/develop/jlmkr.py
./jlmkr.py install
systemd-nspawn is already installed.
The jlmkr command is available.
Done installing jailmaker.

Regarding those nvidia API mismatch errors. You did mention you tried to install the nvidia driver (which you don't have to, and shouldn't, when you have gpu_passthrough_nvidia=1 in your jail config). Jailmaker bind mounts the driver from the host, so it can't have introduced this mismatch.

Perhaps this is just how it is on SCALE, which sounds unlikely but then it would be a SCALE bug. Which version of SCALE do you have? Perhaps upgrade to the latest version and try again with a clean jail using gpu_passthrough_nvidia=1 and don't install any nvidia drivers in it.

[Kailee71]

Thank for your help.
I just installed the dev script to see if it would help with the API mismatch, and the install went smoothly. Created a new Debian test jail with Nvidia passthrough, also succesfully, however, on jail startup...

root@truenas:/mnt/zpool/jailmaker/jails/ubuntu0# jlmkr start debtest
Skipped mounting /dev/nvidia-modeset, it doesn't exist on the host...
Traceback (most recent call last):
File "/usr/local/sbin/jlmkr", line 1365, in
main()
File "/usr/local/sbin/jlmkr", line 1323, in main
start_jail(args.name)
File "/usr/local/sbin/jlmkr", line 423, in start_jail
*shlex.split(config.get("systemd_nspawn_user_args", "")),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/shlex.py", line 315, in split
return list(lex)
^^^^^^^^^
File "/usr/lib/python3.11/shlex.py", line 300, in next
token = self.get_token()
^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/shlex.py", line 109, in get_token
raw = self.read_token()
^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/shlex.py", line 191, in read_token
raise ValueError("No closing quotation")
ValueError: No closing quotation
root@truenas:/mnt/zpool/jailmaker/jails/ubuntu0#

My existing jail with Nvidia passthrough starts up just fine.

Sorry I've no experience with Python, but it looks like it might be trivial fix?

[Jip-Hop]

Can you post the config file you used for this jail? Which version of SCALE are you using?

[Jip-Hop]

Also I recommend to download the nvtop appimage on the TrueNAS host directly. Run it there. If you then also get the "no gpu to monitor" error, then we can conclude that jailmaker isn't causing the error.

[Kailee71]

I'm trying to keep my TrueNAS as vanilla as posssible so I won't install any appimages there - nvtop is not that important. Plus I have a hope it should be able to be made to work in the jail...

[Kailee71]

Ok all seems solved now - I created a new jail with the non-dev jlmkr.py, installed my compute software, and nothing else, and all works as expected, including nvtop. As a heads-up, this is a jammy jail, debian bookwork seems to not provide nvtop through apt...
My apologies for the confusion.