guides/nvidia-jetson/

[giscus[bot]]

guides/nvidia-jetson/

Quick start guide to setting up YOLOv8 on a NVIDIA Jetson device with comprehensive benchmarks.

https://docs.ultralytics.com/guides/nvidia-jetson/

[RasoulZardeyni]

Which one reduce inference time using TensorRT or deepstream?

[pderrenger]

In terms of reducing inference time on your NVIDIA Jetson device, using TensorRT is generally recommended. TensorRT is specifically designed to provide high performance for NVIDIA hardware by optimizing deep learning models for inference. It accelerates models by fusing layers, pruning unused layers, and quantizing variables, among other optimizations. 🚀

DeepStream, on the other hand, is more focused on streaming analytics and can utilize TensorRT-optimized models. For most cases, directly using TensorRT for model optimization is your go-to for minimizing inference times.

Here's a quick example of exporting a YOLO model to TensorRT and running inference, tailored for your convenience:

from ultralytics import YOLO

# Load your YOLO model
model = YOLO('yolov8n.pt')

# Export to TensorRT format
model.export(format='engine')

# Load the TensorRT model
trt_model = YOLO('yolov8n.engine')

# Run inference using the TensorRT model
results = trt_model('your_image_or_video.jpg')

Feel free to explore more about deploying on NVIDIA Jetson with our Quick Start Guide for NVIDIA Jetson. Happy inferencing! 🌟

[RasoulZardeyni]

Thanks!

[pderrenger]

Hey there! 😊 It looks like you're interested in speeding up your inference times. Between TensorRT and DeepStream on NVIDIA Jetson devices, TensorRT typically offers more direct optimization for reducing inference time for individual models by providing lower-level optimizations and quantization options.

DeepStream is a comprehensive framework for building end-to-end video analytics applications, which utilizes TensorRT under the hood for model inference. DeepStream excels in scenarios involving video streams, multi-stream input, and complex pipelines involving multiple AI models and analytics. If your primary goal is solely to minimize inference time for a specific model, direct optimization via TensorRT can provide more focused improvements.

A quick example of exporting a YOLO model to TensorRT:

from ultralytics import YOLO

# Load your YOLO model
model = YOLO('yolov8n.pt')

# Export to TensorRT format
model.export(format='engine')  # This creates 'yolov8n.engine'

# Done! Now you can use 'yolov8n.engine' for faster inference.

However, if your application benefits from a broader video analytics pipeline, incorporating DeepStream into your workflow can optimize and streamline these processes on top of efficiently handling inference with TensorRT optimized models. 🚀

Hope this gives you a clearer picture! If you have more questions, feel free to ask.

[derrick20000]

Hi, for this docker that run on jetpack 4x, I need to use csi camera for my project, and I face a lot of problems about it. Without docker, I can open the camera. With docker, I cant open it although I already add this command --volume /tmp/argus_socket:/tmp/argus_socket. This is my command to activate ultralytics docker:
sudo docker run -it --runtime nvidia
--device /dev/video0
--device /dev/nvhost-ctrl
--device /dev/nvhost-ctrl-gpu
--device /dev/nvhost-prof-gpu
--device /dev/nvmap
--device /dev/nvhost-gpu
--device /dev/nvhost-vic
--device /dev/nvhost-nvdec
--device /dev/nvhost-msenc
--device /dev/nvhost-isp
--device /dev/nvhost-vi
--volume /tmp/argus_socket:/tmp/argus_socket
-v ~/myenv/FYP:/workspace
-e DISPLAY=$DISPLAY
-e QT_X11_NO_MITSHM=1
-v /tmp/.X11-unix:/tmp/.X11-unix
--privileged
ultralytics/ultralytics:latest-jetson-jetpack4

I not sure is the torch and torchvision version that beyond the Jetpack 4 which is 1.11.0 that suppose for Jetpack 5 affect the csi camera. Please help me, thanks.

[glenn-jocher]

@derrick20000 hi there! 😊

Thank you for reaching out with your issue regarding the CSI camera in Docker on JetPack 4.x. It sounds like you're on the right track with your Docker setup, but let's see if we can pinpoint the problem.

Firstly, ensure that you have the latest versions of torch and ultralytics installed. If not, please update them and try again. This can sometimes resolve compatibility issues.

Regarding the CSI camera, the command you provided looks mostly correct. However, there are a few additional steps and checks you can perform:

1. Ensure Camera Access in Docker:
   Make sure the Docker container has the necessary permissions to access the camera. You might need to add more device mappings or permissions. Here's an enhanced version of your Docker run command:

   sudo docker run -it --runtime nvidia \
     --device /dev/video0 \
     --device /dev/nvhost-ctrl \
     --device /dev/nvhost-ctrl-gpu \
     --device /dev/nvhost-prof-gpu \
     --device /dev/nvmap \
     --device /dev/nvhost-gpu \
     --device /dev/nvhost-vic \
     --device /dev/nvhost-nvdec \
     --device /dev/nvhost-msenc \
     --device /dev/nvhost-isp \
     --device /dev/nvhost-vi \
     --volume /tmp/argus_socket:/tmp/argus_socket \
     -v ~/myenv/FYP:/workspace \
     -e DISPLAY=$DISPLAY \
     -e QT_X11_NO_MITSHM=1 \
     -v /tmp/.X11-unix:/tmp/.X11-unix \
     --privileged \
     ultralytics/ultralytics:latest-jetson-jetpack4

2. Check Camera Access Outside Docker:
   Verify that the camera works correctly outside of Docker. You can use tools like v4l2-ctl to check if the camera is accessible:

   v4l2-ctl --list-devices

3. Install Necessary Libraries:
   Ensure that all necessary libraries for camera access are installed within the Docker container. You might need to install v4l-utils and other dependencies:

   sudo apt-get update
   sudo apt-get install v4l-utils

4. Test Camera Access Inside Docker:
   Once inside the Docker container, test if you can access the camera using a simple script or tool. For example, you can use OpenCV to test camera access:

   import cv2
   
   cap = cv2.VideoCapture(0)
   if not cap.isOpened():
       print("Cannot open camera")
       exit()
   while True:
       ret, frame = cap.read()
       if not ret:
           print("Can't receive frame (stream end?). Exiting ...")
           break
       cv2.imshow('frame', frame)
       if cv2.waitKey(1) == ord('q'):
           break
   cap.release()
   cv2.destroyAllWindows()

If the issue persists, please provide a minimum reproducible example of your code and setup. This will help us investigate further. You can find more details on creating a minimum reproducible example here.

For more detailed guidance, please refer to our NVIDIA Jetson Quick Start Guide.

I hope this helps! Let us know how it goes or if you have any other questions. 🚀

[jirivchi1]

Hi, I am using jetson nano with jetpack4 and jetson orin nano with jetpack5.1.3. I am trying to use webcamera inside the docker and with the follow argument it works for jetson nano jetpack4-->

sudo docker run -it --runtime nvidia
--device /dev/video0
--device /dev/nvhost-ctrl
--device /dev/nvhost-ctrl-gpu
--device /dev/nvhost-prof-gpu
--device /dev/nvmap
--device /dev/nvhost-gpu
--device /dev/nvhost-vic
--device /dev/nvhost-nvdec
--device /dev/nvhost-msenc
--device /dev/nvhost-isp
--device /dev/nvhost-vi
--volume /tmp/argus_socket:/tmp/argus_socket
-v ~/myenv/FYP:/workspace
-e DISPLAY=$DISPLAY
-e QT_X11_NO_MITSHM=1
-v /tmp/.X11-unix:/tmp/.X11-unix
--privileged
ultralytics/ultralytics:latest-jetson-jetpack4

My question is: How it change for jetson orin nano? how can I change the last line? " ultralytics/ultralytics:latest-jetson-jetpack4
"

Thanks in advanced

[pderrenger]

@jirivchi1 hi there!

Thank you for reaching out. To run your setup on the Jetson Orin Nano with JetPack 5.1.3, you can modify the Docker command to use the appropriate Docker image for JetPack 5. Here’s how you can adjust the last line of your command:

sudo docker run -it --runtime nvidia \
  --device /dev/video0 \
  --device /dev/nvhost-ctrl \
  --device /dev/nvhost-ctrl-gpu \
  --device /dev/nvhost-prof-gpu \
  --device /dev/nvmap \
  --device /dev/nvhost-gpu \
  --device /dev/nvhost-vic \
  --device /dev/nvhost-nvdec \
  --device /dev/nvhost-msenc \
  --device /dev/nvhost-isp \
  --device /dev/nvhost-vi \
  --volume /tmp/argus_socket:/tmp/argus_socket \
  -v ~/myenv/FYP:/workspace \
  -e DISPLAY=$DISPLAY \
  -e QT_X11_NO_MITSHM=1 \
  -v /tmp/.X11-unix:/tmp/.X11-unix \
  --privileged \
  ultralytics/ultralytics:latest-jetson-jetpack5

This command uses the ultralytics/ultralytics:latest-jetson-jetpack5 image, which is optimized for JetPack 5.x.

For more detailed instructions and additional setup options, you can refer to our Quick Start Guide: NVIDIA Jetson with Ultralytics YOLOv8.

If you encounter any issues or need further assistance, please ensure you are using the latest versions of torch and ultralytics. If the problem persists, providing a minimum reproducible example will help us investigate further. You can find more information on creating a reproducible example here.

Best of luck with your project! 🚀

[jirivchi1]

thanks, it works!

[pderrenger]

@jirivchi1 you're welcome! 😊 I'm glad to hear it worked for you.

If you have any more questions or need further assistance, feel free to ask. For more detailed information on deploying Ultralytics YOLOv8 on NVIDIA Jetson devices, you can refer to our Quick Start Guide: NVIDIA Jetson with Ultralytics YOLOv8.

Happy coding! 🚀

[AgbajeAyomipo]

Good day, as i can see in one of the above pictures, there's 15G of memory available which is quite alright.
i have 3.3G available and i'm detecting in real time, i keep getting a "Low memory" warning after detecting for about 20 minutes. What could the issue be?
I have confirmed that it runs on the gpu.
I have a jetson orin nano.

PS: I intend to detect in real time for hours.

[pderrenger]

Good day!

Thank you for reaching out. The "Low memory" warning you're encountering after 20 minutes of real-time detection on your Jetson Orin Nano could be due to several factors, including memory leaks or inefficient memory management in your code.

Here are a few steps to help diagnose and potentially resolve the issue:

1. Reproducible Example: To better assist you, could you provide a minimum reproducible example of your code? This will help us pinpoint the issue more effectively. You can find guidance on creating a reproducible example here: Minimum Reproducible Example.

2. Update Packages: Ensure you are using the latest versions of all relevant packages, including the Ultralytics YOLOv8 package. Sometimes, updates include important bug fixes and performance improvements.

3. Memory Management: Monitor your memory usage over time to identify any potential leaks. You can use tools like jtop to monitor system resources on your Jetson device:

   sudo apt update
   sudo pip install jetson-stats
   sudo reboot
   jtop

4. Optimize Inference: Consider using TensorRT for optimized inference performance. TensorRT can significantly reduce memory usage and improve inference speed. Here's a quick example of how to export your model to TensorRT and run inference:

   from ultralytics import YOLO
   
   # Load a YOLOv8n PyTorch model
   model = YOLO("yolov8n.pt")
   
   # Export the model to TensorRT format
   model.export(format="engine")  # creates 'yolov8n.engine'
   
   # Load the exported TensorRT model
   trt_model = YOLO("yolov8n.engine")
   
   # Run inference
   results = trt_model("https://ultralytics.com/images/bus.jpg")

5. Best Practices: Follow best practices for maximizing performance on NVIDIA Jetson devices:

   • Enable MAX Power Mode:

     sudo nvpmodel -m 0

   • Enable Jetson Clocks:

     sudo jetson_clocks

For more detailed guidance, you can refer to our Quick Start Guide: NVIDIA Jetson with Ultralytics YOLOv8.

I hope this helps! Please let us know if you have any further questions or need additional assistance.

[AgbajeAyomipo]

import ultralytics
from ultralytics import YOLO

model = YOLO("yolov8n.pt")

Dependencies:
* torch==2.1.0a0+41361538.nv23.6
* ultralytics==8.2.38

Now, in the other discussion i made, i was asked to export the model:
i ran the following code:


model.export(format = "engine", workspace = 2, int8 = True)

Error Message after running this:
[06/26/2024-12:24:43] [TRT] [W] Tactic Device request: 150MB Available: 149MB. Device memory is insufficient to use tactic.
[06/26/2024-12:24:43] [TRT] [W] Skipping tactic 2 due to insufficient memory on requested size of 150 detected for tactic 0x000000000000001e.

Thank You.

[pderrenger]

Hello,

Thank you for providing the details of your setup and the error message you're encountering. It looks like the issue is related to insufficient memory for the TensorRT tactic during the export process.

Here are a few steps to help resolve this issue:

1. Reduce Workspace Size: The workspace parameter specifies the maximum workspace size in GiB for TensorRT optimizations. You might want to try reducing this value further to fit within your available memory. For example:

   model.export(format="engine", workspace=1, int8=True)

2. Check Memory Usage: Monitor your system's memory usage to ensure there are no other processes consuming significant memory. You can use the jtop tool for this:

   sudo apt update
   sudo pip install jetson-stats
   sudo reboot
   jtop

3. Enable MAX Power Mode and Jetson Clocks: Ensure your Jetson device is running at its maximum performance settings:

   sudo nvpmodel -m 0
   sudo jetson_clocks

4. Verify Latest Versions: Make sure you are using the latest versions of all relevant packages. Sometimes, updates include important bug fixes and performance improvements.

If the issue persists, providing a minimum reproducible example can be very helpful for further diagnosis. You can find guidance on creating a reproducible example here: Minimum Reproducible Example.

Feel free to reach out if you have any further questions or need additional assistance. 😊

[jirivchi1]

I tried ultralytics for jetson orin nano "Start without Docker" and I am getting the follow error:

(env3.8) jetson-orin@jetsonorin-desktop:~/Desktop/gpu$ python3
Python 3.8.10 (default, Nov 22 2023, 10:22:35)
[GCC 9.4.0] on linux
Type “help”, “copyright”, “credits” or “license” for more information

Traceback (most recent call last):
File “”, line 1, in
File “/home/jetson-orin/Desktop/gpu_cocacola/env3.8/lib/python3.8/site-packages/ultralytics/init.py”, line 10, in
from ultralytics.data.explorer.explorer import Explorer
File “/home/jetson-orin/Desktop/gpu_cocacola/env3.8/lib/python3.8/site-packages/ultralytics/data/init.py”, line 3, in
from .base import BaseDataset
File “/home/jetson-orin/Desktop/gpu_cocacola/env3.8/lib/python3.8/site-packages/ultralytics/data/base.py”, line 17, in
from ultralytics.data.utils import FORMATS_HELP_MSG, HELP_URL, IMG_FORMATS
File “/home/jetson-orin/Desktop/gpu_cocacola/env3.8/lib/python3.8/site-packages/ultralytics/data/utils.py”, line 19, in
from ultralytics.nn.autobackend import check_class_names
File “/home/jetson-orin/Desktop/gpu_cocacola/env3.8/lib/python3.8/site-packages/ultralytics/nn/init.py”, line 3, in
from .tasks import (
File “/home/jetson-orin/Desktop/gpu_cocacola/env3.8/lib/python3.8/site-packages/ultralytics/nn/tasks.py”, line 10, in
from ultralytics.nn.modules import (
File “/home/jetson-orin/Desktop/gpu_cocacola/env3.8/lib/python3.8/site-packages/ultralytics/nn/modules/init.py”, line 20, in
from .block import (
File “/home/jetson-orin/Desktop/gpu_cocacola/env3.8/lib/python3.8/site-packages/ultralytics/nn/modules/block.py”, line 8, in
from ultralytics.utils.torch_utils import fuse_conv_and_bn
File “/home/jetson-orin/Desktop/gpu_cocacola/env3.8/lib/python3.8/site-packages/ultralytics/utils/init.py”, line 47, in
TORCHVISION_VERSION = importlib.metadata.version(“torchvision”) # faster than importing torchvision
File “/usr/lib/python3.8/importlib/metadata.py”, line 530, in version
return distribution(distribution_name).version
File “/usr/lib/python3.8/importlib/metadata.py”, line 503, in distribution
return Distribution.from_name(distribution_name)
File “/usr/lib/python3.8/importlib/metadata.py”, line 177, in from_name
raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: torchvision

I am seeing is not only my problem and there is a post in NVIDIA forum: https://forums.developer.nvidia.com/t/pytorch-installation-on-jetson-agx-orin/292846

help: last line of installation: python3 setup.py install --user
is saying this kinf of warning --> torchvision/csrc/ops/cuda/ps_roi_pool_kernel.cu(283)

[pderrenger]

Hello @jirivchi1,

Thank you for reaching out and providing detailed information about the issue you're encountering. It appears that the error is related to the torchvision package not being found, which is a common issue on Jetson devices due to the ARM64 architecture.

To resolve this, please follow these steps to ensure that torch and torchvision are correctly installed for your Jetson Orin Nano:

1. Uninstall the existing torch and torchvision packages:

   pip uninstall torch torchvision

2. Install the required dependencies:

   sudo apt-get install -y libopenblas-base libopenmpi-dev libjpeg-dev zlib1g-dev

3. Download and install the compatible PyTorch wheel:

   wget https://developer.download.nvidia.com/compute/redist/jp/v512/pytorch/torch-2.1.0a0+41361538.nv23.06-cp38-cp38-linux_aarch64.whl -O torch-2.1.0a0+41361538.nv23.06-cp38-cp38-linux_aarch64.whl
   pip install torch-2.1.0a0+41361538.nv23.06-cp38-cp38-linux_aarch64.whl

4. Clone and install torchvision from source:

   git clone https://github.com/pytorch/vision torchvision
   cd torchvision
   git checkout v0.16.2
   python3 setup.py install --user

5. Ensure onnxruntime-gpu is installed correctly:

   wget https://nvidia.box.com/shared/static/zostg6agm00fb6t5uisw51qi6kpcuwzd.whl -O onnxruntime_gpu-1.17.0-cp38-cp38-linux_aarch64.whl
   pip install onnxruntime_gpu-1.17.0-cp38-cp38-linux_aarch64.whl

6. Reinstall numpy to the compatible version:

   pip install numpy==1.23.5

After completing these steps, please try running your script again. This should resolve the torchvision import error and allow you to proceed with your project.

If the issue persists, please ensure you are using the latest versions of all packages and provide a minimum reproducible example as outlined here to help us diagnose further.

For more detailed guidance, you can refer to our NVIDIA Jetson Quick Start Guide.

Best regards and happy coding! 🚀

Ultralytics Team

[sophiapeckner]

Hello! I'm running YOLOv8 on my Jetson Nano JetPack 4.x. I took the Docker approach.
I've run yolo predict on bus.jpg ('https://ultralytics.com/images/bus.jpg') using TorchScript and TensorRT format which I exported using yolo export:

yolo predict model=yolov8n.torchscript source=bus.jpg
Speed: 93.6ms preprocess, 215ms inference, 151.7ms postprocess per image at shape (1, 3, 640, 640)

yolo predict model=yolov8n.engine source=bus.jpg
Speed: 7243.4ms preprocess, 3195.6ms inference, 11259.1ms postprocess per image at shape (1, 3, 640, 640)

These results are unexpected. You'd think TensorRT/.engine would be faster but in fact it's much slower. I'm wondering if this should raise any red flags? And if so, how might I proceed? I also ask because I've been having a lot of trouble exporting YOLOv8 on the Nano so just wanted to be on the lookout for any potential problems so that I can catch them early on. Thanks in advance!

[glenn-jocher]

@sophiapeckner it seems unusual that your TensorRT model is performing slower than the TorchScript model. This could be due to several factors, including the export process or runtime environment. First, ensure you are using the latest versions of the Ultralytics package and JetPack. Additionally, verify that TensorRT is properly installed and configured. You might also want to check if the Jetson Nano is running in MAXN mode and that Jetson clocks are enabled. For further details, you can refer to our NVIDIA Jetson guide. If the issue persists, please provide more details about your setup.

[sophiapeckner]

Ultralytics package is 8.2.57 (I checked in Docker container)
JetPack version is 4.6.5 which is the highest version supported by the Nano
TensorRT is installed; could you provide more guidance on how to check if it's properly configured?
MAXN mode and Jetson clocks are enabled
I looked at Ultralytic's Docker Support Guide. When I run nvidia-smi I get command not found. According to this forum:

Ubuntu 18.04 ships the nvidia-smi binary in the nvidia-utils-390 package. Unfortunately this isn't built for the arm64 architecture

I have Ubuntu 18.04 and getting a newer image that supports Ubuntu 22.04 isn't an option. Do you have any thoughts on how I should proceed?

[glenn-jocher]

Thank you for the detailed information. Given your setup, the significant slowdown with TensorRT is indeed unexpected. First, ensure that the TensorRT engine was correctly optimized during export. You can try re-exporting the model with the simplify and dynamic arguments to see if it improves performance:

yolo export model=yolov8n.pt format=engine simplify=True dynamic=True

Additionally, verify that TensorRT is properly installed and configured by running a simple TensorRT sample provided in the NVIDIA TensorRT documentation. If the issue persists, consider checking for any potential bottlenecks in your Docker setup or system configuration. For further guidance, you can refer to our NVIDIA Jetson guide.

[MadhavKrishna1]

when i run this command on Jetson Xavier NX (Jetpack 5.1.2)
"sudo apt install -y libjpeg-dev zlib1g-dev
git clone https://github.com/pytorch/vision torchvision
cd torchvision
git checkout v0.16.2
python3 setup.py install --user"
i get this error:
"python3 setup.py install --user
Traceback (most recent call last):
File "setup.py", line 9, in
import torch
ModuleNotFoundError: No module named 'torch'"

What should i do?

[MadhavKrishna1]

i got it solved, actually i missed one download and a installation was missing

"wget https://developer.download.nvidia.com/compute/redist/jp/v512/pytorch/torch-2.1.0a0+41361538.nv23.06-cp38-cp38-linux_aarch64.whl -O torch-2.1.0a0+41361538.nv23.06-cp38-cp38-linux_aarch64.whl
pip install torch-2.1.0a0+41361538.nv23.06-cp38-cp38-linux_aarch64.whl".

[glenn-jocher]

@MadhavKrishna1 glad to hear you resolved the issue! Installing the correct PyTorch version is crucial for compatibility on Jetson devices. If you encounter any further issues or need additional guidance, feel free to refer to our NVIDIA Jetson Quick Start Guide. It provides detailed steps and troubleshooting tips for setting up YOLOv8 on Jetson devices.

[MadhavKrishna1]

WARNING ⚠️ TensorRT requires GPU export, automatically assigning device=0
Ultralytics YOLOv8.2.60 🚀 Python-3.8.10 torch-2.1.0a0+41361538.nv23.06 CUDA:0 (Xavier, 14898MiB)

I am getting this warning when i ran these code
"from ultralytics import YOLO

Load a YOLOv8n PyTorch model

model = YOLO("yolov8n.pt")

Export the model

model.export(format="engine") "

Even though i have my torch cuda enabled.
torch.cuda.is_available() shows output True.

[pderrenger]

@MadhavKrishna1 the warning indicates that TensorRT requires GPU export, which is why it automatically assigns device=0. Even though torch.cuda.is_available() returns True, the export process specifically needs to ensure that the GPU is utilized. Please verify that your NVIDIA Jetson device is correctly set up with the latest JetPack and CUDA drivers. If the issue persists, try running the export command within a Docker container as described in the NVIDIA Jetson setup guide. This can help ensure all dependencies are correctly configured.

[DJT777]

Is there any intention to update the documentation for Jetpack 6?

https://docs.ultralytics.com/guides/nvidia-jetson

[glenn-jocher]

Thank you for your interest. Currently, there are no immediate plans to update the documentation specifically for JetPack 6. However, we continually review and update our resources to ensure compatibility with the latest technologies. Please keep an eye on our documentation page for any future updates. If you have any specific issues or questions, feel free to ask!

[knowee97]

Hi, I plan on using the docker that'll run on jetpack 4.x and would need to use a CSI camera (IMX519) for my project. As it is my first time using a docker, are there any videos that will demonstrate a step process on using a docker on a Jetson Nano?

[glenn-jocher]

@knowee97 for a step-by-step guide on using Docker with Jetson Nano, I recommend watching the video linked in the Quick Start Guide: "How to Setup NVIDIA Jetson with Ultralytics YOLOv8." This guide covers Docker setup and usage on Jetson devices, including camera integration. For detailed instructions, visit the Ultralytics documentation at https://docs.ultralytics.com/guides/nvidia-jetson/. This should help you get started with your CSI camera and Docker on JetPack 4.x. If you encounter specific issues, feel free to ask!

[knowee97]

I was able to run the Docker under the "Run on JetPack 4.x" section on my Jetson Nano Dev Kit. After mounting files from the Jetson to the docker, I wasn't able to access my CSI camera (IMX159) using a quick test code from previous comments. It worked as usual outside the docker, however wouldn't in the docker. I checked using "ls -l /dev/video0" in the docker and detected it. This is my command to activate the ultralytics docker:

sudo docker run -it --ipc=host
--runtime nvidia
--device /dev/video0:/dev/video0
-v /home/noe/Desktop/pyProg:/usr/src/ultralytics/myProg
ultralytics/ultralytics:latest-jetson-jetpack4

[glenn-jocher]

It sounds like you've set up the Docker container correctly. However, accessing the CSI camera within Docker can sometimes be tricky. Ensure that you have the necessary permissions and that the camera is correctly mapped. You might also want to try adding --privileged to your Docker run command to grant the container additional privileges. If the issue persists, please verify that all necessary drivers and libraries for the camera are available within the Docker environment. For further guidance, refer to the detailed steps in the Ultralytics Jetson Guide.

[knowee97]

An update to using the docker was made below. However, I still haven't been able to access the cameras (CSI & Webcam) even when using --privileged. Outside the docker works great and inside the docker is recognized, yet no able to access it and output frame. What are some possibilities I can try and take?

sudo docker run -it --ipc=host --runtime=nvidia
--gpus all
--device /dev/video0:/dev/video0
--device /dev/video1:/dev/video1
-v /path/to/code:/usr/src/ultralytics/myProg
-v /path/to/Models:/usr/src/ultralytics/Models
--privileged
ultralytics/ultralytics:latest-jetson-jetpack4

[glenn-jocher]

It seems you've set up the Docker container correctly, but accessing the CSI and webcam cameras can be tricky. Ensure that all necessary drivers and libraries for the cameras are available within the Docker environment. You might also try running the container with --privileged and --device /dev/video0:/dev/video0 --device /dev/video1:/dev/video1 to map the devices correctly. Additionally, verify that the container has access to the necessary video group permissions. If the issue persists, consider checking the Docker logs for any specific errors related to camera access. For more detailed guidance, refer to the Ultralytics Jetson Guide.

[Faker-T1-NO1]

hi , i have error: torchvision 0.16.2 requires torch==2.1.2, but you have torch 2.1.0a0+41361538.nv23.6 which is incompatible.
how to slove this prolem

[glenn-jocher]

@Faker-T1-NO1 to resolve the compatibility issue between torchvision and torch, you need to ensure both packages are compatible with each other. Uninstall the current versions and install the compatible versions as follows:

1. Uninstall the existing packages:

   pip uninstall torch torchvision

2. Install the compatible versions:

   sudo apt-get install -y libopenblas-base libopenmpi-dev
   wget https://developer.download.nvidia.com/compute/redist/jp/v512/pytorch/torch-2.1.0a0+41361538.nv23.06-cp38-cp38-linux_aarch64.whl
   pip install torch-2.1.0a0+41361538.nv23.06-cp38-cp38-linux_aarch64.whl
   sudo apt install -y libjpeg-dev zlib1g-dev
   git clone https://github.com/pytorch/vision torchvision
   cd torchvision
   git checkout v0.16.2
   python3 setup.py install --user

This should resolve the version mismatch. For more details, refer to the NVIDIA Jetson setup guide.

[Faker-T1-NO1]

Yes, I follow this but have this error Vào Th 6, 2 thg 8, 2024 lúc 12:11 Glenn Jocher ***@***.***> đã viết:

…

@Faker-T1-NO1 <https://github.com/Faker-T1-NO1> to resolve the compatibility issue between torchvision and torch, you need to ensure both packages are compatible with each other. Uninstall the current versions and install the compatible versions as follows: 1. Uninstall the existing packages: pip uninstall torch torchvision 2. Install the compatible versions: sudo apt-get install -y libopenblas-base libopenmpi-dev wget https://developer.download.nvidia.com/compute/redist/jp/v512/pytorch/torch-2.1.0a0+41361538.nv23.06-cp38-cp38-linux_aarch64.whl pip install torch-2.1.0a0+41361538.nv23.06-cp38-cp38-linux_aarch64.whl sudo apt install -y libjpeg-dev zlib1g-dev git clone https://github.com/pytorch/vision torchvisioncd torchvision git checkout v0.16.2 python3 setup.py install --user This should resolve the version mismatch. For more details, refer to the NVIDIA Jetson setup guide <https://docs.ultralytics.com/guides/nvidia-jetson/>. — Reply to this email directly, view it on GitHub <#9942 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BKIS5BJS3DHXAF74PQRH5FDZPMIINAVCNFSM6AAAAABGAK2IGGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMRRHE2DQNQ> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[glenn-jocher]

@Faker-T1-NO1 it seems like you're encountering an error despite following the instructions. Please ensure you are using the latest versions of the packages. If the issue persists, try reinstalling PyTorch and Torchvision as described. For further guidance, you can refer to the detailed setup instructions in the NVIDIA Jetson setup guide. If the problem continues, please provide more details about the error message for further assistance.

[MadhavKrishna1]

When i try to run "results = model.benchmarks(data="coco8.yaml", imgsz=640)"
I get this error:

Traceback (most recent call last):
File "1.py", line 10, in
results=model.benchmarks(data="coco8.yaml",imgsz=640)
File "/home/nvidia/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1614, in getattr
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'YOLO' object has no attribute 'benchmarks'

How do i resolve it, I am doing it on Jetson Xavier NX?

[glenn-jocher]

@MadhavKrishna1 the error you're encountering suggests that the benchmarks method might not be available in your current version of the Ultralytics package. Please ensure you are using the latest version by running pip install --upgrade ultralytics. If the issue persists, refer to the NVIDIA Jetson setup guide for additional configuration steps.

[Vidhul-S]

Hello , for docker version and jetpack4x on jetson nano this is my command to run

sudo docker run -it --runtime nvidia \
  --device /dev/video0 \
  --device /dev/nvhost-ctrl \
  --device /dev/nvhost-ctrl-gpu \
  --device /dev/nvhost-prof-gpu \
  --device /dev/nvmap \
  --device /dev/nvhost-gpu \
  --device /dev/nvhost-vic \
  --device /dev/nvhost-nvdec \
  --device /dev/nvhost-msenc \
  --device /dev/nvhost-isp \
  --device /dev/nvhost-vi \
  --volume /tmp/argus_socket:/tmp/argus_socket \
  -v ~/myenv/FYP:/workspace \
  -e DISPLAY=$DISPLAY \
  -e QT_X11_NO_MITSHM=1 \
  -v /tmp/.X11-unix:/tmp/.X11-unix \
  --privileged \
  ultralytics/ultralytics:latest-jetson-jetpack4

I am not getting a display output even though the code is running and detection results r printed on the terminal

I assume that

-e DISPLAY=$DISPLAY

the above line is the one to support display and i have included it
I am using the nano with a display and in stand-alone mode

How to fix that what changes should i make to the command OR any other changes needed

[glenn-jocher]

@Vidhul-S to resolve the display issue, ensure that X11 forwarding is enabled on your Jetson Nano. You can try adding --net=host to your Docker command to allow the container to use the host's network settings. This often resolves display issues with GUI applications. If the problem persists, please check the Docker and Jetson documentation for additional troubleshooting steps.

[Vidhul-S]

Thanks a lot but that command didn't work but your suggestion to look at docker guides help and i got it running
After running the docker run command and while being attached to the container terminal
run the below code before you run your own model

apt update
apt install x11-apps

include --net=host in your run command and add xhost +local:root
So your run command should look like this

xhost +local:root
sudo docker run -it --runtime nvidia \
  --name test1 \
  --device /dev/video0 \
  --device /dev/nvhost-ctrl \
  --device /dev/nvhost-ctrl-gpu \
  --device /dev/nvhost-prof-gpu \
  --device /dev/nvmap \
  --device /dev/nvhost-gpu \
  --device /dev/nvhost-vic \
  --device /dev/nvhost-nvdec \
  --device /dev/nvhost-msenc \
  --device /dev/nvhost-isp \
  --device /dev/nvhost-vi \
  --volume /tmp/argus_socket:/tmp/argus_socket \
  --volume /home/intern/models \
  --net="host" \
  --env="DISPLAY" \
  --env="QT_X11_NO_MITSHM=1" \
  --volume="/tmp/.X11-unix:/tmp/.X11unix:rw" \
  --privileged \
  ultralytics/ultralytics:latest-jetson-jetpack4 \

If you run the model with this setup you will get your gui output

[glenn-jocher]

Great to hear you got it working! Adding --net=host and running xhost +local:root are indeed necessary for GUI output. Thanks for sharing your solution! For more details, you can refer to the NVIDIA Jetson guide.

[mleibi]

Hello,
i did the installation with Jetpack 5.1.3 (on Nvidia Jetson Orin NX) and got everything installed. However when i try to rebuild the test, i get an error in the line model.export(format="engine")
 The error is that it doesn't find tensorRT. However i have TensorRT installed, version 8.5.2 and cuda 11.4 as it came with Jepack 5.1.3.
The error message is:
WARNING ⚠️ TensorRT requires GPU export, automatically assigning device=0
Ultralytics YOLOv8.2.82 🚀 Python-3.8.19 torch-2.1.0a0+41361538.nv23.06 CUDA:0 (Orin, 7451MiB)
YOLOv8n summary (fused): 168 layers, 3,151,904 parameters, 0 gradients, 8.7 GFLOPs

PyTorch: starting from 'yolov8n.pt' with input shape (1, 3, 640, 640) BCHW and output shape(s) (1, 84, 8400) (6.2 MB)

ONNX: starting export with onnx 1.16.2 opset 17...
====== Diagnostic Run torch.onnx.export version 2.1.0a0+41361538.nv23.06 =======
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

ONNX: export success ✅ 1.4s, saved as 'yolov8n.onnx' (12.2 MB)
requirements: Ultralytics requirement ['tensorrt>7.0.0,<=10.1.0'] not found, attempting AutoUpdate...
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│exit code: 1
╰─>[6 lines of output]
Traceback (most recent call last):
File "", line 2, in
File "", line 34, in
File "/tmp/pip-install-5zo24jso/tensorrt-cu12_cf5a1445e4bf4c0e891ae82bb592b914/setup.py", line 67, in
raise RuntimeError("TensorRT currently only builds wheels for x86_64 processors")
RuntimeError: TensorRT currently only builds wheels for x86_64 processors
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Retry 1/2 failed: Command 'pip install --no-cache-dir "tensorrt>7.0.0,<=10.1.0" ' returned non-zero exit status 1.
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
exit code: 1
╰─>[6 lines of output]
Traceback (most recent call last):
File "", line 2, in
File "", line 34, in
File "/tmp/pip-install-wf15bax9/tensorrt-cu12_374073278f144e508931e40962dec7c9/setup.py", line 67, in
raise RuntimeError("TensorRT currently only builds wheels for x86_64 processors")
RuntimeError: TensorRT currently only builds wheels for x86_64 processors
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Retry 2/2 failed: Command 'pip install --no-cache-dir "tensorrt>7.0.0,<=10.1.0" ' returned non-zero exit status 1.
requirements: ❌ Command 'pip install --no-cache-dir "tensorrt>7.0.0,<=10.1.0" ' returned non-zero exit status 1.
TensorRT: export failure ❌ 7.9s: No module named 'tensorrt'

Please tell me if you need any more information. I'd really appreciate some help
Matthias

[glenn-jocher]

Hi Matthias,

It seems like the issue might be related to the TensorRT Python bindings. Ensure you have the python3-libnvinfer and python3-libnvinfer-dev packages installed. If the issue persists, try reinstalling these packages and rebooting your device. For more detailed steps, please refer to our NVIDIA Jetson guide.

[mleibi]

Hi Glenn,
Thanks for your answer. the python3-libnvinfer and python3-libnvinfer-dev packages were installed and i flashed the jetson again and they are installed. i did use conda and virtual environement. is it possible that this is the problem? the virtual environement not seeing the two above mentioned libraries?
Thank you Matthias

[glenn-jocher]

Hi Matthias, using Conda environments can sometimes cause issues with system libraries like TensorRT. Try running your setup outside of Conda to ensure the environment can access these libraries. Let me know if this resolves the issue!

[mleibi]

Hi Glenn,
it did resolve it. I'm using venv instead of conda now and this solves the problem.
Thank you Matthias

[glenn-jocher]

Great to hear that switching to venv resolved the issue, Matthias. If you have any more questions, feel free to ask!

[CH-Echo]

I would like to ask why I deployed my self trained m model on Jetson Orin NX, and it takes 50ms for frame inference and 30ms for frame inference in FP16

[glenn-jocher]

@CH-Echo the difference in inference times between 50ms and 30ms for FP16 on your Jetson Orin NX could be due to the efficiency of FP16 precision, which reduces computation and memory usage. For optimal performance, ensure you're using TensorRT and have enabled MAX Power Mode and Jetson Clocks. For more details, please refer to the NVIDIA Jetson guide.

[NidhoggBo]

My device is a jetson nano, can I use the gpu version of pytorch after I pull the docker image?

[glenn-jocher]

@NidhoggBo yes, you can use the GPU version of PyTorch on a Jetson Nano after pulling the Docker image. Ensure you have the appropriate JetPack version installed for compatibility. For detailed setup, please refer to the NVIDIA Jetson guide on our documentation site.

[kaplam]

Hello !

I am using YOLOv8 with Docker (*) on Jetson Nano Jetpack 4.6.1 [L4T 32.7.1]

I had problem with Python to display the results. Here is an example:

from ultralytics import YOLO

Load a model

model = YOLO("yolov8n.pt") # pretrained YOLOv8n model

Run batched inference on a list of images

results = model(["image1.jpg", "image2.jpg"]) # return a list of Results objects

Process results list

for result in results:
boxes = result.boxes # Boxes object for bounding box outputs
masks = result.masks # Masks object for segmentation masks outputs
keypoints = result.keypoints # Keypoints object for pose outputs
probs = result.probs # Probs object for classification outputs
obb = result.obb # Oriented boxes object for OBB outputs
result.show() # display to screen
result.save(filename="result.jpg") # save to disk

I get:

0: 640x480 4 persons, 1 bus, 1 stop sign, 258.6ms
1: 640x480 4 persons, 1 bus, 1 stop sign, 258.6ms
Speed: 421.8ms preprocess, 258.6ms inference, 506.5ms postprocess per image at shape (1, 3, 640, 480)

but nothing is displayed.
Please can you help me solve this problem ? I have tried various ways, but none good.
Thanks

[glenn-jocher]

@kaplam it seems like the result.show() function isn't displaying the images on your Jetson Nano. This might be due to display limitations in your Docker setup. Try saving the results to disk and viewing them outside the container. If you need further assistance, please refer to the NVIDIA Jetson guide for more details.

[NidhoggBo]

How do I use the Jetson.GPIO in a jetpack4 docker image

[glenn-jocher]

@NidhoggBo to use Jetson.GPIO in a JetPack 4 Docker image, ensure the GPIO library is installed within the container. You might need to map the necessary device files and permissions from the host to the container. For detailed steps, refer to NVIDIA's official documentation on GPIO usage in Docker.

[mohammed01997]

If I work on Sahi algorithm with Yolo 8, will these things apply to it? NVIDIA A100:

FP32 Inference: ~0.52 ms / image
FP16 Inference: ~0.34 ms / image
INT8 Inference: ~0.28 ms / image
Slight reduction in mAP with INT8 precision, but significant improvement in speed.
Consumer GPUs (e.g., RTX 3080):

FP32 Inference: ~1.06 ms / image
FP16 Inference: ~0.62 ms / image
INT8 Inference: ~0.52 ms / image Note that the image size is MB and its dimensions are 12000 * 9000

[mohammed01997]

If the number of slices is 432 with a size of 640 * 640, is every slice considered an image?

[glenn-jocher]

@mohammed01997 yes, each slice of 640x640 is considered an individual image when using the SAHI algorithm with YOLOv8. If you have further questions, feel free to ask!

[mohammed01997]

If I work on Sahi algorithm with Yolo 8, will these things apply to it? NVIDIA A100:

FP32 Inference: ~0.52 ms / image
FP16 Inference: ~0.34 ms / image
INT8 Inference: ~0.28 ms / image
Slight reduction in mAP with INT8 precision, but significant improvement in speed.
Consumer GPUs (e.g., RTX 3080):

FP32 Inference: ~1.06 ms / image
FP16 Inference: ~0.62 ms / image
INT8 Inference: ~0.52 ms / image Note that the image size is MB and its dimensions are 12000 * 9000

[glenn-jocher]

The performance metrics you mentioned for the Sahi algorithm with YOLOv8 on NVIDIA A100 and consumer GPUs should generally apply, but specific results may vary based on your setup and optimizations. For detailed guidance on deploying YOLOv8 on NVIDIA Jetson, please refer to the Ultralytics Jetson Guide.

[Noman-Nosher]

How can I deploy YOLO11 model on Jetson TX2?

[glenn-jocher]

@Noman-Nosher to deploy YOLO11 on a Jetson TX2, please follow the detailed guide available at Ultralytics YOLO on NVIDIA Jetson. This resource covers installation and optimization steps for Jetson devices.

[Noman-Nosher]

Jetson TX2 support Jetpack 4. But for ultralytics package installation, Jetpack 5 is required as it comes with Python 3.8. I have implemented YOLOv8 on Jetson TX2 but with workarounds which leads to performance issues afterwards.

Is this compatible with Jetson TX2 to implement YOLO11 or Hardware Update is necessary for this?
Thank you

[glenn-jocher]

YOLO11 can be deployed on Jetson TX2, but it requires JetPack 5 for optimal performance. If you're experiencing issues with JetPack 4, consider upgrading your hardware or using workarounds cautiously, as they may lead to performance degradation. For detailed setup instructions, please refer to the Ultralytics Jetson Guide.

[QingLea]

Hi, Jetpack 6.1 has been released, the yolo model cannot be trained on this platform, which gives an error.

ImportError: libcudnn.so.8: cannot open shared object file: No such file or directory

I assume it is because my cudnn version is updated to 9.3.0 along with Jetpack 6.1.

[QingLea]

Jetpack 6.1
dpkg -l | grep cudnn

ii  cudnn                                      9.3.0-1                                     arm64        NVIDIA CUDA Deep Neural Network library (cuDNN)
ii  cudnn9                                     9.3.0-1                                     arm64        NVIDIA CUDA Deep Neural Network library (cuDNN)
ii  cudnn9-cuda-12                             9.3.0.75-1                                  arm64        NVIDIA cuDNN for CUDA 12
ii  cudnn9-cuda-12-6                           9.3.0.75-1                                  arm64        NVIDIA cuDNN for CUDA 12.6
ii  libcudnn9-cuda-12                          9.3.0.75-1                                  arm64        cuDNN runtime libraries for CUDA 12.6
ii  libcudnn9-dev-cuda-12                      9.3.0.75-1                                  arm64        cuDNN development headers and symlinks for CUDA 12.6
ii  libcudnn9-samples                          9.3.0.75-1                                  all          cuDNN samples
ii  libcudnn9-static-cuda-12                   9.3.0.75-1                                  arm64        cuDNN static libraries for CUDA 12.6

Jetpack 6
dpkg -l | grep cudnn

ii  libcudnn8                                  8.9.4.25-1+cuda12.2                               arm64        cuDNN runtime libraries
ii  libcudnn8-dev                              8.9.4.25-1+cuda12.2                               arm64        cuDNN development libraries and headers

[glenn-jocher]

@QingLea it seems like the issue is related to the cuDNN version mismatch. Please ensure that the correct version of cuDNN is installed and linked properly. You might need to adjust your environment variables or create symbolic links to the correct libcudnn.so file. For detailed setup instructions, please refer to our NVIDIA Jetson guide.

[QingLea]

Thank you for your response. I'd like to highlight that Jetpack has recently been updated to Version 6.1, which now includes cuDNN9 instead of cuDNN8. Due to this update, the current YOLO implementation doesn't work as expected with the new cuDNN version.

Could you please update the setup manual and the YOLO library dependencies to reflect this change? This would help avoid similar issues for other users as well.

[glenn-jocher]

Thank you for highlighting this issue. We'll review the compatibility with Jetpack 6.1 and update the setup instructions as needed. For now, you might consider reverting to a compatible cuDNN version. Please check our NVIDIA Jetson guide for any updates.

[Noman-Nosher]

I am currently working with the NVIDIA Jetson TX2, which has 8 GB of LPDDR4 memory and 32 GB of eMMC storage. My device runs JetPack 4, and I am deploying YOLOv8 on it.

While setting up, I encountered a "disk space 0" error when attempting to run Docker commands, which I suspect is due to storage constraints. Could you advise on the recommended free space required for running a pre-built Docker image of JetPack 4 with YOLOv8?

Additionally, I have configured YOLOv8 in a CUDA-enabled virtual environment. However, I am experiencing issues linking TensorRT to this environment, which is preventing me from converting the YOLOv8 .pt model to TensorRT’s .engine format.

Could you recommend solutions or steps for successfully performing this model conversion, either within the virtual environment or using Docker?

[glenn-jocher]

@Noman-Nosher for the "disk space 0" error, ensure you have at least 10 GB of free space for Docker operations. Regarding TensorRT conversion, verify that TensorRT is correctly installed and linked in your environment. You might find additional guidance in our NVIDIA Jetson setup guide.

[Noman-Nosher]

Yes, TensorRT is correctly installed within python3.6 version which is pre-installed with jetpack. but while linking that tensorRT from local environment to virtual environment which is python 3.8, is creating issue and not converting the .pt file to .engine. I am also attaching the error screenshoot.

Error:
(myenv) noman@ubuntu:~/Python-3.8.12$ yolo export model=yolovén.pt format=onnx
WARNING f\Python>=3.10 is required, but Python==3.8.12 is currently installed
Ultralytics 8.3.20 # Python-3.8.12 torch-1.11.8a0+gitbc2c6ed CPU (ARMv8 Processor rev 3 (v8l))
YOLOVSN summary (fused): 168 layers, 3,151,904 parameters, © gradients, 8.7 GFLOPS
PyTorch: starting fron 'yolovan.pt' with input shape (1, 3, 640, 640) BCH4 and output shape(s) (1, 84, 8400) (6.2 MB)
requirenents: Ultralytics requirement ['onnxslim'] not found, attempting AutoUpdate...
Collecting onnxslin
Downloading onnxslin-0.1.35-py3-none-any.whi.netadata (3.0 kB)
Requirement already satisfied: onnx in ./myenv/lib/python3.s/site-packages (from onnxslin) (1.17.0)
Requirement already satisfied: synpy in ./nyenv/lib/python3.8/site-packages (from onnxslim) (1.13.3)
Requirement already satisfied: packaging in ./myenv/1ib/python3.8/site-packages (from onnxslin) (24.1)
Requirement already satisfied: numpy>=1.20 in ./myenv/lib/python3.8/site-packages (from onnx->onnxslim) (1.24.4)
Requirement already satisfied: protobuf>=3.26.2 in ./myenv/lib/python3.8/site-packages (from onnx->onnxslin) (5.28.3)
Requirement already satisfied: mpmath<1.d,>=1.1.6 in ./nyenv/lib/python3.8/site-packages (from sympy->onnxsiin) (1.3.0)
Downloading onnxslin-0.1.35-py3-none-any.whl (140 k8)
Installing collected packages: onnxslin
Successfully installed onnxslin-6.1.35
requirements: AutoUpdate success y/ 5.5s, installed 1 package: ['onnxslim']
requirencnts: fARestart runtime or rerun command for updates to take effect
OMNX: starting export with onnx 1.17.0 opset 14...
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.

[glenn-jocher]

It seems like the issue is with Python version compatibility. Ensure TensorRT is installed for Python 3.8, or consider using Python 3.6 where TensorRT is already working. For Docker, ensure at least 10 GB of free space. More details can be found in our NVIDIA Jetson guide.

[tunahsu]

Is there a way to verify how many YOLOv8n/YOLO11n models the Jetson AGX Orin 64G can run simultaneously, or does it require testing by incrementally adding models one at a time?

[glenn-jocher]

@tunahsu to determine how many YOLOv8n/YOLO11n models the Jetson AGX Orin 64G can run simultaneously, it's best to test incrementally by adding models one at a time. For more detailed guidance, please refer to our NVIDIA Jetson guide.

[swyangGntsolution]

Hello!! ... I want to set up Ultralytics and PyTorch in JetPack 6.1 on a Jetson AGX Orin.

I followed your guide, but it’s not working well.

1. Update packages list, install pip, and upgrade to latest:
   """"""""""""""""""""""""""""""""""""""""""""""
   sudo apt update
   sudo apt install python3-pip -y
   pip install -U pip
   """"""""""""""""""""""""""""""""""""""""""""""

2. Install Ultralytics with optional dependencies:
   """"""""""""""""""""""""""""""""""""""""""""""
   pip install ultralytics[export]
   """"""""""""""""""""""""""""""""""""""""""""""
   At this stage, Ultralytics works well.

3. Reboot the device:
   """"""""""""""""""""""""""""""""""""""""""""""
   sudo reboot
   """"""""""""""""""""""""""""""""""""""""""""""

4. Install PyTorch and Torchvision:
   """"""""""""""""""""""""""""""""""""""""""""""
   sudo apt-get install libopenmpi-dev libopenblas-base libomp-dev -y
   pip install https://github.com/ultralytics/assets/releases/download/v0.0.0/torch-2.3.0-cp310-cp310-linux_aarch64.whl
   pip install https://github.com/ultralytics/assets/releases/download/v0.0.0/torchvision-0.18.0a0+6043bc2-cp310-cp310-linux_aarch64.whl
   """"""""""""""""""""""""""""""""""""""""""""""

But here’s where the issue starts!

pip’s dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torchvision 0.20.1 requires torch==2.5.1, but you have torch 2.3.0, which is incompatible.

After installing torch, torchvision and trying to run Ultralytics, I get this same error again.

ImportError: libcudnn.so.8: cannot open shared object file: No such file or directory

[glenn-jocher]

@swyangGntsolution it seems like there might be a mismatch in the PyTorch and torchvision versions you're using. Please ensure that both packages are compatible with each other and your JetPack version. Additionally, the libcudnn error suggests that the CUDA libraries might not be properly installed. You can refer to the NVIDIA Jetson setup guide for detailed instructions on resolving these issues: https://docs.ultralytics.com/guides/nvidia-jetson/.

[swyangGntsolution]

Thank you for your response. However, I initially installed ultralytics following the guide, and it worked perfectly fine. The issues began only after I installed PyTorch and torchvision as per the guide's instructions under 'Native Installation' for JetPack 6.x. This led to a libcudnn-related error and a version mismatch, which prevented ultralytics from running.

Could there be an issue with the recommended PyTorch and torchvision versions in the guide, or is there something I should adjust in the installation steps to avoid these compatibility problems?

[swyangGntsolution]

Oh! I’m currently using Anaconda. Do I need to set this up without Anaconda on Jetson?"

[swyangGntsolution]

I'm sorry. I'll look into it further. Please close this session.

[glenn-jocher]

It seems like the issue might be related to version compatibility between PyTorch and torchvision. Please ensure that both packages are compatible with your JetPack version. Additionally, using Anaconda on Jetson devices can sometimes lead to conflicts; it's recommended to follow the native installation guide without Anaconda for better compatibility. For further details, you can refer to the NVIDIA Jetson setup guide.

[KwCCCC]

Hi. I wonder if the model exported using tensorRT is the same as the normal model object. For example,

model = YOLO("yolo11n.pt")

Export the model to TensorRT with DLA enabled (only works with FP16 or INT8)

model.export(format="engine", device="dla:0", half=True) # dla:0 or dla:1 corresponds to the DLA cores

Load the exported TensorRT model

trt_model = YOLO("yolo11n.engine")

Run inference

results = trt_model("https://ultralytics.com/images/bus.jpg")

I wonder if trt_model in the code above is ultralytics.engine.model.model class. If it is the same, is it correct to use methods such as load, save, predict, and train in this model object? If not, please let me know the page with specific information about this. Thanks!

[glenn-jocher]

@KwCCCC the trt_model is indeed an instance of the ultralytics.engine.model.Model class, allowing you to use methods like predict. However, training and saving are not applicable for TensorRT models. For more details, please refer to the Ultralytics documentation.

[vinura]

Hi is the guide updated for Jetpack 6.1 ? it feels like torch version are for jetpack 6.0 (please remove tested for jetpack 6.x) since the new jepack 6.1 comes with CUDA 12.6. where as the torch given on the guide is for 12.2

[glenn-jocher]

@vinura thank you for pointing this out. The guide currently reflects compatibility based on JetPack 6.0, and we will update it to align with JetPack 6.1 and CUDA 12.6 as soon as possible. For now, please consult NVIDIA's PyTorch for Jetson resource for Torch versions compatible with CUDA 12.6 or manually compile Torch if needed.

[ooobelix]

Hey there!
I'm using NVIDIA Jetson Xavier NX with JEtPack 5.1-b147 and following "Run on JetPack 5.x" instructions.
First issue appears with "pip install torch-2.1.0a0+41361538.nv23.06-cp38-cp38-linux_aarch64.whl":

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed.
This behaviour is the source of the following dependency conflicts.
ultralytics 8.3.40 requires torchvision>=0.9.0, which is not installed.
Successfully installed torch-2.1.0a0+41361538.nv23.6

Since Torch seems to be installed successfully, I try to install Torchvision v0.16.2 with "python3 setup.py install --user" but an new issue:

[...]
OSError: libcufft.so.10: cannot open shared object file: No such file or directory
[...]
ValueError: libcublas.so.*[0-9] not found in the system path[...]
[...]

Then I have tried to install with "pip install ." according this link.

[...]
/tmp/pip-build-env-fa3m3lsj/overlay/lib/python3.8/site-packages/torch/_subclasses/functional_tensor.py:258: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at /pytorch/csrc/utils/tensor_numpy.cpp:84.)
  cpu = _conversion_method_template(device=torch.device("cpu"))
<string>:10: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
/tmp/pip-build-env-fa3m3lsj/overlay/lib/python3.8/site-packages/setuptools/_distutils/cmd.py:111: SetuptoolsDeprecationWarning: bdist_wheel.universal is deprecated
[...]
      subprocess.CalledProcessError: Command '['which', 'aarch64-linux-gnu-g++']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for torchvision
Failed to build torchvision
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (torchvision)

Some information:

$ pip list |grep "torch\|ultra\|numpy"
numpy               1.23.1
torch               2.1.0a0+41361538.nv23.6
ultralytics         8.3.40
ultralytics-thop    2.0.12

$ dpkg -l |grep -i cuda
ii  cuda-cudart-11-4                     11.4.298-1                            arm64        CUDA Runtime native Libraries
ii  cuda-nvrtc-11-4                      11.4.300-1                            arm64        NVRTC native runtime libraries
ii  cuda-toolkit-11-4-config-common      11.4.298-1                            all          Common config package for CUDA Toolkit 11.4.
ii  cuda-toolkit-11-config-common        11.4.298-1                            all          Common config package for CUDA Toolkit 11.
ii  cuda-toolkit-config-common           11.4.298-1                            all          Common config package for CUDA Toolkit.
ii  libcudnn8                            8.6.0.166-1+cuda11.4                  arm64        cuDNN runtime libraries
ii  libnvinfer-plugin8                   8.5.2-1+cuda11.4                      arm64        TensorRT plugin libraries
ii  libnvinfer8                          8.5.2-1+cuda11.4                      arm64        TensorRT runtime libraries
ii  libnvonnxparsers8                    8.5.2-1+cuda11.4                      arm64        TensorRT ONNX libraries
ii  libnvparsers8                        8.5.2-1+cuda11.4                      arm64        TensorRT parsers libraries
ii  nvidia-l4t-cuda                      35.2.1-20230124153320                 arm64        NVIDIA CUDA Package

Can you help me? Thanks!

[glenn-jocher]

@ooobelix it seems the issues stem from compatibility between the installed CUDA toolkit and the required libraries for PyTorch and Torchvision. Please verify that your Jetson setup and CUDA environment match the requirements for the versions of PyTorch (2.1.0) and Torchvision you’re installing. Specifically:

• Ensure libraries like cufft, cublas, and others required by Torchvision are installed or appropriately linked.
• You may want to revisit NVIDIA's PyTorch for Jetson page to confirm compatibility between the versions of JetPack, CUDA, and the specific PyTorch/Torchvision builds.
• Reinstall numpy to version 1.23.5 after any installations, as it might be impacted by onnxruntime-gpu.

For further guidance, check the Jetson setup guide for YOLO.

[ooobelix]

You are right, it working well with:

$ pip list |grep "torch\|ultra\|numpy"   
numpy               1.23.1                                    
torch               2.0.0+nv23.5                              
torchvision         0.15.1a0+42759b1                          
ultralytics         8.3.40                                    
ultralytics-thop    2.0.12                                    
                                                              
[notice] A new release of pip is available: 23.3.2 -> 24.3.1  
[notice] To update, run: python3 -m pip install --upgrade pip

[glenn-jocher]

Thank you for confirming that the configuration works well now! It seems you've effectively resolved the previous issues by aligning the appropriate PyTorch and Torchvision versions. Let us know if you encounter any further questions or require additional assistance! For reference, the detailed setup process on NVIDIA Jetson can also be reviewed here.

[menkeyi001]

jetson nx information:
Platform                                  
Machine: aarch64
System: Linux                             Hardware  
                                            Model: NVIDIA Orin NX Developer 
KitDistribution:Ubuntu 20.04 focal          699-level Part Number:699-13767-0000-300 P.1
Release:5.10.104-tegra                      P-Number: p3767-0800
Python: 3.9.20                              Module:NVIDIA Jetson Orin NX(16GB ram)
                                            Soc: tegra23x
                                            CUDA Arch BIN: 8.7
Libraries                                   Codename:P3768
CUDA: 11.4.315                              L4T: 35.3.1
CuDNN:8.6.0.166                             Jetpack:5.1.1
TensorRT:8.5.2.2
VPI:2.2.7
Vulkan:1.3.204
Opency:4.5.4 with CUDA: NO



docker pull 
ultralytics/ultralytics:latest-jetson-jetpack5
sudo docker pull ultralytics/ultralytics:latest-jetson-jetpack5 
sudo docker run -it --ipc=host --runtime=nvidia ultralytics/ultralytics:latest-jetson-jetpack5 



root@ubuntu:/ultralytics# yolo export model=yolo11n.pt format=engine 
WARNING ⚠️ TensorRT requires GPU export, automatically assigning device=0
Ultralytics 8.3.51 🚀 Python-3.8.10 torch-2.4.1 
Traceback (most recent call last):
  File "/usr/local/bin/yolo", line 8, in <module>
    sys.exit(entrypoint())
  File "/ultralytics/ultralytics/cfg/__init__.py", line 972, in entrypoint
    getattr(model, mode)(**overrides)  # default args from model
  File "/ultralytics/ultralytics/engine/model.py", line 738, in export
    return Exporter(overrides=args, _callbacks=self.callbacks)(model=self.model)
  File "/ultralytics/ultralytics/engine/exporter.py", line 225, in __call__
    self.device = select_device("cpu" if self.args.device is None else self.args.device)
  File "/ultralytics/ultralytics/utils/torch_utils.py", line 192, in select_device
    raise ValueError(
ValueError: Invalid CUDA 'device=0' requested. Use 'device=cpu' or pass valid CUDA device(s) if available, i.e. 'device=0' or 'device=0,1,2,3' for Multi-GPU.

torch.cuda.is_available(): False
torch.cuda.device_count(): 0
os.environ['CUDA_VISIBLE_DEVICES']: None
See https://pytorch.org/get-started/locally/ for up-to-date torch install instructions if no CUDA devices are seen by torch.


Is there something wrong with my docker image version? Not compatible with my device？

[menkeyi001]

I use this docker image version touch and can use cuda normally ngc.nju.edu.cn/nvidia/l4t-pytorch r35.2.1-pth2.0-py3

[glenn-jocher]

@menkeyi001 it seems that the Docker image you mentioned might not have proper CUDA or GPU support for your setup. Please confirm that you are using the correct ultralytics/ultralytics:latest-jetson-jetpack5 image, which is compatible with JetPack 5.1.1. Additionally, ensure that Jetson-specific settings like nvpmodel and jetson_clocks are configured for maximum performance. If issues persist, you may need to manually verify the CUDA and PyTorch installation versions inside the container. You can refer to this guide for more details: https://docs.ultralytics.com/guides/nvidia-jetson/.