Realsense camera ROS drivers with the Jetson TX2

[haashimrehan]

Documenting progress on getting Realsense D435 and T265 working with ROS Melodic on the Jetson TX2 with L4T 32.7.3.

simply installing the binaries of realsense2_camera and librealsense2 doesn't work when pointclouds are enabled, as ROS topics don't publish any messages (at least reliably).

JetsonUtilites output:
NVIDIA quill
L4T 32.7.3 [ JetPack 4.6.3 ]
Ubuntu 18.04.6 LTS
Kernel Version: 4.9.299-tegra
CUDA NOT_INSTALLED
CUDA Architecture: 6.2
OpenCV version: 4.1.1
OpenCV Cuda: NO
CUDNN: 8.2.1.32
TensorRT: 8.2.1.9
Vision Works: 1.6.0.501
VPI: 1.2.3
Vulcan: 1.2.70

[haashimrehan]

built librealsense2 from source with the help of:
https://github.com/UofA-Robomasters/installLibrealsenseTX2
Building Realsense ROS from source results in catkin build errors due to cv_bridge not liking opencv4

[haashimrehan]

ros-perception/vision_opencv#320 (comment)

[haashimrehan]

after following these steps (cloning a repo that supports opencv4):
git clone https://github.com/OAkyildiz/vision_opencv
checkout melodic
catkin_make

I could build successfully, however the cameras don't launch:

D435:

[ WARN] [1683955511.153558303]: Device 1/1 failed with exception: failed to set power state
 13/05 01:25:11,151 ERROR [547323105664] (handle-libusb.h:51) failed to open usb interface: 0, error: RS2_USB_STATUS_ACCESS
 13/05 01:25:11,152 ERROR [547620979072] (sensor.cpp:572) acquire_power failed: failed to set power state

T265:

No RealSense devices were found!
 13/05 01:24:32,204 ERROR [547742544256] (handle-libusb.h:51) failed to open usb interface: 0, error: RS2_USB_STATUS_ACCESS
 13/05 01:24:32,204 ERROR [547742544256] (tm-boot.h:42) Failed to open T265 zero interface

[haashimrehan]

Forgot to set the udev rules, cameras are now detected!

Instructions are here:
IntelRealSense/realsense-ros#1408 (comment)

After I solved that I tried to run the launch file again but started having issues with space, I was getting really suspicious and with the help of chatgpt we figured out that there was a logfile called uvcdynctrl-udev.log that was a whole 13GB! needless to say that needed to go.

with enough free space to start the node, I once again try to start the launch file this time faced with this error:
Hardware Notification:Depth stream start failure,1.68396e+12,Error,Hardware Error

a quick google search leads me to this:
IntelRealSense/realsense-ros#1533 (comment)

which makes me suspect I might need to downgrade the firmware of the D435.

[haashimrehan]

connected the D435 to my laptop to downgrade the firmware with:

rs-fw-update -f ./Signed_Image_UVC_5_12_15_50.bin
firmware was obtained from: https://dev.intelrealsense.com/docs/firmware-releases

didnt help with the error

[haashimrehan]

ran out of space again which is odd. that same log file was 9GB. I deleted it again and started monitoring the storage on the Jetson. As soon as I plug in the camera I can see space running out on the Jetson in real time.

I should probably read whats in that log file...

UPDATE: it spams this

[libwebcam] Warning: The driver behind device video0 has a slightly buggy implementation
  of the V4L2_CTRL_FLAG_NEXT_CTRL flag. It does not return the next higher
  control ID if a control query fails. A workaround has been enabled.

[haashimrehan]

I'm now able to use both cameras in ROS, just cant view the raw pointcloud from the D435 without it crashing after a few seconds. I believe this has something to do with the TX2's kernel. To get around that we'll just convert the depth image to a laserscan. due to this issue you can't set enable_pointcloud to true.

I had to build librealsense2 from source following this. This follows the Ubuntu install instructions and not the one for the Jetson because its outdated. I also didnt apply any kernel patches.

I cloned the realsense ROS 1 branch into my ROS ws. I've added it as a submodule in the repo which can be downloaded with git submodule init and git submodule update

this time I installed the binary for cv_bridge (ros-melodic-cv-bridge) and manually changed a file to fix the opencv4 issue on the jetson: see here

I also fixed the log spam with sudo apt-get purge uvcdynctrl

[haashimrehan]

To avoid rebuilding the source files for librealsense2 on the other TX2 (due to limited time and space constraints), I used checkinstall instead of make install to build a custom mmrt-librealsense2.deb binary

The checkinstall command calls the make install command. It monitors the files that are installed and creates a binary package from them. It also installs the binary package with the Linux package manager. I then transfer this binary package to the other TX2 and use sudo dpkg --install mmrt-librealsense2_20230515-1_arm64.deb to install the package.

Another benefit of this method is that it allows for a clean uninstall with:
sudo apt remove mmrt-librealsense2

[haashimrehan]

Disable USB autosuspend on the TX2 (requires reboot):
sudo sed -i '$s/$/ usbcore.autosuspend=-1/' /boot/extlinux/extlinux.conf

[haashimrehan]

UPDATE: on a fresh Jetpack install I ended up installing the latest binaries for librealsense2 and realsense2 ros. We will continue with this configuration on the rover since we are using the latest Jetpack that doesn't support the official kernel modifications provided by intel and it is simpler and less of a headache to debug if we face issues. without the kernel modifications we see minimal performance benefits, and I found that adjusting camera parameters through rqt_reconfigure had a greater affect (things like lowering the exposure, and jpeg compression over remote ROS).

L4T 32.7.3 [ JetPack 4.6.3 ]
Ubuntu 18.04.6 LTS
Kernel Version: 4.9.299-tegra
CUDA 10.2.300
CUDA Architecture: 6.2
OpenCV version: 4.1.1
OpenCV Cuda: NO
CUDNN: 8.2.1.32
TensorRT: 8.2.1.9
Vision Works: 1.6.0.501
VPI: 1.2.3
Vulcan: 1.2.70

[haashimrehan]

This is relevant for getting a pointcloud published.

IntelRealSense/realsense-ros#2646 (comment)

it requires ros-melodic-rgbd-launch to be installed and essentially combines the depth image with the RGB image to generate a pointcloud.

[emcauliffe]

Re-added git submodule for realsense-ros package with fix to the realsense2_camera package so that the built package uses the local CUDA version as per comment here

IntelRealSense/realsense-ros#2326 (comment)

[emcauliffe]

Upon building opencv, librealsense, and realsense-ros from source, errors were now occurring with the PCIe card.

To resolve this, we disabled PCIe ASPM on the jetson (see wiki: https://github.com/MacRover/Rover/wiki/Disable-aspm)

Now, there is an error that occurs when running only the t265.

jetson-1 kernel: xhci_hcd 0000:04:00.0: WARN Event TRB for slot 1 ep 4 with no TDs queued?

This appears to be a bug in the linux kernel that has not been back-ported by nvidia to the tegra kernel running on the TX2. It does not appear to impact the performance of the cameras in any way.

http://billauer.co.il/blog/2019/11/xhci-controller-bug-warning/

https://lore.kernel.org/lkml/[email protected]/