Tensorrt python bindings

[tirankas]

Hello,
Does the Nvidia Yocto supports the tensorrt python binding?
I am looking for the tensorrt.py file and cannot find it.
Do I need to select a specific package for the binding to occur?

My image currently built with tensorrt package (image insta append) but I don't see any tensorrt.so file in the build outputs, an I missing anything?

Thanks

Thanks

[madisongh]

Unfortunately, no. The tensorrt package includes a .so file for its Python bindings that is linked against Python 3.6 and is not compatible with any other Python version. Only the C/C++ libraries (libnvinfer.so and friends) are available.

I know that NVIDIA has open-sourced some parts of TensorRT (see here), which looks like it include Python support, but I have no idea whether any of it's usable or adaptable to our builds.

[tirankas]

Thanks Matt!
I have indeed seen this tensorrt open source release and from 5 minutes looking I believe it should be pretty easily adopted to the tegra demo distro.
It's basically relying on the pybind and python to build the so.
I will give it a shot and will update.

[tirankas]

Hi Matt,
I opened a ticker for Nvidia since I face some issues compiling for Yocto:
NVIDIA/TensorRT#1254

Basically I created the recipe and fails during compilation (15% success...)

I believe this recipe should end up in Yocto once completed, this is a major blocker for having python bindings for the Tensorrt, up to you guys...

Thanks

[ichergui]

Hey @tirankas

Could you please share your recipe, maybe we could help you.
Are you sure about the dependencies in your recipe ?

Did you try to build the NVIDIA/TensorRT with ubuntu / Jetpack 4.5 ? if no, please try it and let us know about the result.

[tirankas]

"Did you try to build the NVIDIA/TensorRT with ubuntu / Jetpack 4.5 ? if no, please try it and let us know about the result"

You mean build on the Jetson device directly (not cross compiling)?
I did a week ago, had some issues, I will try again and let you know.

[ichergui]

Yes, on the jetson device (No cross compilation) using ubuntu from jetpack 4.5

[tirankas]

Sure, here it is (Some paths are using recipe-sysroot explicitely, I will fix this later):

SUMMARY = "Tensorrt OSS"

LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"

DEPENDS += "python3 cmake-native python3-pybind11 tensorrt protobuf-c-native protobuf-c libcublas cuda-cudart cuda-nvrtc libglvnd cudnn"

COMPATIBLE_MACHINE = "(tegra)"

inherit cmake pkgconfig cuda

SRC_URI = "
file://TensorRT-release-7.1.zip
"

S = "${WORKDIR}/TensorRT-release-7.1"
B = "${S}"

CUDA_VERSION="10.2"
CUDA_PATH = "${S}/../recipe-sysroot/usr/local/cuda-${CUDA_VERSION}"

TRT_GLOBAL_INC_DIR ?= "${S}/../recipe-sysroot/usr/include"

EXTRA_OECMAKE_append = '
-DCUDA_VERSION="10.2"
-DCUDA_PATH="${S}/../recipe-sysroot/usr/local/cuda-${CUDA_VERSION}"
-DCUDA_ROOT="${CUDA_PATH}"
-DCMAKE_TOOLCHAIN_FILE="${S}/cmake/toolchains/cmake_aarch64.toolchain"
-DTRT_LIB_DIR="${S}/../recipe-sysroot/usr/lib"
-DCUDA_NVCC_FLAGS="${CUDA_NVCC_FLAGS}"
-DTRT_GLOBAL_INC_DIR="${TRT_GLOBAL_INC_DIR}"
-DGPU_ARCHS="72"
'

[ichergui]

I will try it later when I have time and I will let you know

[tirankas]

Thanks, appreciate your help

[madisongh]

There were a couple of problems with this.

First, the python bindings aren't in their 7.1 release branch, so this recipe wouldn't supply what you're looking for. They didn't add the python bindings until the 2021.02 release. The good news there, though, is it wasn't too hard to adapt just the python bindings to work with the tensorRT 7.1.3 as we get it from NVIDIA. Take a look at PR #712 and see if that works for you.

Second, building the entire open-source TensorRT package would be a much bigger deal, and would require a lot more work. But since it was Python bindings you were after, what's in #712 might be enough.

[tirankas]

Hi Matt,
One last thing before we close this discussion: At the end of the day, the above recipe will end up creating a python bindings for the tensorrt based on the host machine python version.
In reality, we use the tensorrt python in a docker container. The container coming in from Nvidia are based on a different python version and hence the host binding will not work.
What am I missing here?

Thanks

[madisongh]

For use in the container, you'd need to install bindings built for whatever version of python resides in the container. If the container has python 3.6, installing the NVIDIA-provided python3-libnvinfer_7.1.3-1+cuda10.2_arm64.deb from JetPack into the container should work, I would think.

[tirankas]

Thanks Matt, this doesnt work due to other reasons (failed during the install with Invalid cross-device link).

But, in theory, the python bindings we do on the host side is part of the runtime available to a container?
Moreover, if the host and container used the same python version, would the host bindings for the tensorrt would satisfy the container too?

[madisongh]

We'd have to update the recipe to inherit container-runtime-csv and set CONTAINER_CSV_FILES to get the files exported (you could add a bbappend to do this for your builds). As long as the container's python version matches that of the underlying host, it should work.

[tirankas]

What does this means:
"inherit container-runtime-csv and set CONTAINER_CSV_FILES to get the files exported"

You mean without this the tensorrt bindings is not exported?

[tirankas]

I think I get it.
Is below correct (besides the inherit):
CONTAINER_CSV_FILES = "${libdir}/*.so"

Should I add anything else?

[tirankas]

Sorry, meant CONTAINER_CSV_FILES = "${libdir}/*"

[madisongh]

I think that would cover it, although it's probably too generic, and might interfere with the python installation in the container.

CONTAINER_CSV_FILES = "${PYTHON_SITEPACKAGES_DIR}/*"

would probably be better.