Hardware codec not usable in docker container but works in container R36.3

[hsuanguo]

Testing demo-image-full R36.3 commit e470442d70133ad06acd931f57d91f52d16c3439

The hardware codec works fine at host, but in container I kept getting error,
docker is deepstream 7.0

docker run -it --rm --runtime nvidia --net=host --privileged  nvcr.io/nvidia/deepstream:7.0-samples-multiarch bash

Then run

gst-launch-1.0 uridecodebin uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! fakesink

kept getting this error

Failed to query video capabilities: Inappropriate ioctl for device
libv4l2: error getting capabilities: Inappropriate ioctl for device

Error getting capabilities for device '/dev/v4l2-nvdec': It isn't a v4l2 driver. Check if it is a v4l1 driver.
/dvs/git/dirty/git-master_linux/3rdparty/gst/gst-v4l2/gst-v4l2/v4l2_calls.c(107): gst_v4l2_get_capabilities (): /GstPipeline:pipeline0/GstURIDecodeBin:uridecodebin0/GstDecodeBin:decodebin0/nvv4l2decoder:nvv4l2decoder0:
system error: Inappropriate ioctl for device

Wonder if anyone has any idea how this can be solved, I feel like that some libraries are not properly mapped into the container?

[madisongh]

Try nvcr.io/nvidia/deepstream-l4t:7.0-samples-multiarch instead. That's the one I used for testing.

[hsuanguo]

Hi, Thanks for the reply, but it's actually the exact same image:

root@p3768-0000-p3767-0000:~# docker images
REPOSITORY                      TAG                     IMAGE ID       CREATED       SIZE
nvcr.io/nvidia/deepstream       7.0-samples-multiarch   861dc6c94f75   6 weeks ago   6.28GB
nvcr.io/nvidia/deepstream-l4t   7.0-samples-multiarch   861dc6c94f75   6 weeks ago   6.28GB

So I still got the same error. Did it work in your test?

This was how I built the image:

source ./setup-env --machine p3768-0000-p3767-0000 
bitbake demo-image-full

[madisongh]

I hadn't noticed that they were actually the same image. And I hadn't noticed this particular problem you're seeing, because I had run the installation and setup scripts in the container - one of which installed some gstreamer plugins that got used instead of the nvv4l2decoder plugin.

I've tracked down the problem to the symlink handling in our patch to nvidia-container-toolkit. I've pushed a fix for that, so if you update to scarthgap latest, you should find it works better. Specifically, you should see symlinks show up under /usr/lib/aarch64-linux-gnu/libv4l/plugins/nv inside the container. Those missing V4L plugins, due to the symlinks not getting created, were causing the problem with the nvv4l2 plugins.

[hsuanguo]

Thanks for the update, but I am still encountering the same error

Couldn't set nvv4l2decoder0 to READY:
Could not get/set settings from/on resource.
Error getting capabilities for device '/dev/v4l2-nvdec': It isn't a v4l2 driver. Check if it is a v4l1 driver.
/dvs/git/dirty/git-master_linux/3rdparty/gst/gst-v4l2/gst-v4l2/v4l2_calls.c(107): gst_v4l2_get_capabilities (): /GstPipeline:pipeline0/GstURIDecodeBin:uridecodebin0/GstDecodeBin:decodebin0/nvv4l2decoder:nvv4l2decoder0:
system error: Inappropriate ioctl for device

Build info

meta                 = "HEAD:0efe15c5ba29ba4d8599b5d151e6e8ffbf9b253d"
meta-tegra           = "HEAD:ac0b6028de3c062114238f8fe5c5fe5c81ba3cfc"
meta-oe              
meta-python          
meta-networking      
meta-filesystems     = "HEAD:4a7bb77f7ebe0ac8be5bab5103d8bd993e17e18d"
meta-virtualization  = "HEAD:37c06acf58f9020bccfc61954eeefe160642d5f3"
meta-tegra-community = "HEAD:6e9998fb6f4c53d52309746aeec3f7ffafbba494"
meta-tegra-support   
meta-demo-ci  

files under /usr/lib/aarch64-linux-gnu/libv4l/plugins/nv

lrwxrwxrwx 1 root root 34 Jun 17 10:40 libv4l2_nvargus.so -> ../../../nvidia/libv4l2_nvargus.so
lrwxrwxrwx 1 root root 44 Jun 17 10:40 libv4l2_nvcuvidvideocodec.so -> ../../../nvidia/libv4l2_nvcuvidvideocodec.so
lrwxrwxrwx 1 root root 39 Jun 17 10:40 libv4l2_nvvideocodec.so -> ../../../nvidia/libv4l2_nvvideocodec.so

It can be tested using the following if you have other decoders installed

This fails:

gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! fakesink

The following sw decoding works fine in container:

gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! avdec_h264 ! fakesink

[hsuanguo]

Sorry, it actually works fine, I needed to use

docker run -it --rm --runtime nvidia --net=host nvcr.io/nvidia/deepstream:7.0-samples-multiarch bash

instead of

docker run -it --rm --runtime nvidia --net=host --privileged  nvcr.io/nvidia/deepstream:7.0-samples-multiarch bash

Do you know why the --privileged matters here?

Never mind, I found that it's just sometimes it works, sometimes it does not.....

For example, this time it does not work

root@p3768-0000-p3767-0000:~# docker run -it --rm --runtime nvidia --net=host nvcr.io/nvidia/deepstream:7.0-samples-multiarch bash

==========
== CUDA ==
==========

CUDA Version 12.2.12

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

root@p3768-0000-p3767-0000:/opt/nvidia/deepstream/deepstream-7.0# gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! fakesink
<deleted all the warnings as they are not relevant...>
Setting pipeline to PAUSED ...
Failed to query video capabilities: Inappropriate ioctl for device
libv4l2: error getting capabilities: Inappropriate ioctl for device
ERROR: from element /GstPipeline:pipeline0/nvv4l2decoder:nvv4l2decoder0: Error getting capabilities for device '/dev/v4l2-nvdec': It isn't a v4l2 driver. Check if it is a v4l1 driver.
Additional debug info:
/dvs/git/dirty/git-master_linux/3rdparty/gst/gst-v4l2/gst-v4l2/v4l2_calls.c(107): gst_v4l2_get_capabilities (): /GstPipeline:pipeline0/nvv4l2decoder:nvv4l2decoder0:
system error: Inappropriate ioctl for device
ERROR: pipeline doesn't want to preroll.
ERROR: from element /GstPipeline:pipeline0/nvv4l2decoder:nvv4l2decoder0: Could not initialize supporting library.
Additional debug info:
../gst-libs/gst/video/gstvideodecoder.c(2909): gst_video_decoder_change_state (): /GstPipeline:pipeline0/nvv4l2decoder:nvv4l2decoder0:
Failed to open decoder
ERROR: pipeline doesn't want to preroll.
Failed to set pipeline to PAUSED.
Setting pipeline to NULL ...
Freeing pipeline ...
root@p3768-0000-p3767-0000:/opt/nvidia/deepstream/deepstream-7.0# gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! fakesink
Setting pipeline to PAUSED ...
Failed to query video capabilities: Inappropriate ioctl for device
libv4l2: error getting capabilities: Inappropriate ioctl for device
ERROR: from element /GstPipeline:pipeline0/nvv4l2decoder:nvv4l2decoder0: Error getting capabilities for device '/dev/v4l2-nvdec': It isn't a v4l2 driver. Check if it is a v4l1 driver.
Additional debug info:
/dvs/git/dirty/git-master_linux/3rdparty/gst/gst-v4l2/gst-v4l2/v4l2_calls.c(107): gst_v4l2_get_capabilities (): /GstPipeline:pipeline0/nvv4l2decoder:nvv4l2decoder0:
system error: Inappropriate ioctl for device
ERROR: pipeline doesn't want to preroll.
ERROR: from element /GstPipeline:pipeline0/nvv4l2decoder:nvv4l2decoder0: Could not initialize supporting library.
Additional debug info:
../gst-libs/gst/video/gstvideodecoder.c(2909): gst_video_decoder_change_state (): /GstPipeline:pipeline0/nvv4l2decoder:nvv4l2decoder0:
Failed to open decoder
ERROR: pipeline doesn't want to preroll.
Failed to set pipeline to PAUSED.
Setting pipeline to NULL ...
Freeing pipeline ...
root@p3768-0000-p3767-0000:/opt/nvidia/deepstream/deepstream-7.0#

But if I exit the container, the start a new container, then there is a chance that it might work

ERROR: pipeline doesn't want to preroll.
Failed to set pipeline to PAUSED.
Setting pipeline to NULL ...
Freeing pipeline ...
root@p3768-0000-p3767-0000:/opt/nvidia/deepstream/deepstream-7.0# exit
exit
root@p3768-0000-p3767-0000:~# docker run -it --rm --runtime nvidia --net=host nvcr.io/nvidia/deepstream:7.0-samples-multiarch bash

==========
== CUDA ==
==========

CUDA Version 12.2.12

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

root@p3768-0000-p3767-0000:/opt/nvidia/deepstream/deepstream-7.0# gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! fakesink

<deleted all the warnings as they are not relevant...>

Setting pipeline to PAUSED ...
Opening in BLOCKING MODE
Pipeline is PREROLLING ...
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
Redistribute latency...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:00:03.600543689
Setting pipeline to NULL ...
Freeing pipeline ...
root@p3768-0000-p3767-0000:/opt/nvidia/deepstream/deepstream-7.0#

You can see that the newly created container works just fine, they are running exactly the same docker run command, the gst-launch are the same as well.

I also find that /usr/lib/aarch64-linux-gnu/libv4l/plugins/nv sometime can have less symlinks, though it's not relevant to this issue...

root@p3768-0000-p3767-0000:/opt/nvidia/deepstream/deepstream-7.0# ll /usr/lib/aarch64-linux-gnu/libv4l/plugins/nv
total 8
drwxr-xr-x 2 root root 4096 Jun 17 11:29 ./
drwxr-xr-x 1 root root 4096 Jun 17 11:29 ../
lrwxrwxrwx 1 root root   34 Jun 17 11:29 libv4l2_nvargus.so -> ../../../nvidia/libv4l2_nvargus.so
lrwxrwxrwx 1 root root   39 Jun 17 11:29 libv4l2_nvvideocodec.so -> ../../../nvidia/libv4l2_nvvideocodec.so

And another file which is not created correctly to container:

Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libgstnvvideoconvert.so': libnvdsbufferpool.so.1.0.0: cannot open shared object file: No such file or directory

[madisongh]

If you could turn on debug logging in the config.toml file (uncomment the debug = line in the [nvidia-container-runtime] section, and change the log-level to be "debug" instead of "info") and generate logs for the working and non-working cases, that might help in figuring out the problem.

[hsuanguo]

which and how can I check the logs?

[hsuanguo]

nvidia-container-runtime.log
Got the log now, here I ran the container for 2 times:
The first time the v4l2 failed to open, and the second time(last) it works fine, maybe you can spot something unusual?

[madisongh]

I do see some minor differences between the two runs - there's a function to resolve a chain of symbolic links that isn't returning the chain entries in the same order each time, which might explain the inconsistency you're seeing between runs. I'll see if I can fix that.

[madisongh]

OK, I've updated the patch again to eliminate the problem caused by the unordered values returned by that function. Try updating to scarthgap latest again, hopefully I got it right this time.

[hsuanguo]

Seems to be working fine now, thanks for all your help!