Videostreaming delay issues on an ARM64 build of QGC for Raspberry Pi

[jack-obrien]

Hi all,

I have managed to compile the master branch of QGC on a Pi 5 8G. I have done this in docker to maintain dependencies - including Gstreamer 1.18 as reccomended by the QGC docs.

My issue is that video streaming comes in slow motion. I am using a standard RTP H264 video stream over UDP. Here is the pipeline used by a 2nd rasp pi to stream the video over UDP:

/usr/bin/libcamera-vid -t 0 -n --inline --height 720 --width 1280 --framerate 60 --rotation 180 --profile high --level 4.2 --bitrate 1000000 --libav-format h264 -o - \
      | gst-launch-1.0 -v fdsrc fd=0 \
      ! h264parse config-interval=-1 \
      ! queue max-size-time=100000000 \
           ! rtph264pay pt=96 mtu=1024 config-interval=-1 aggregate-mode=zero-latency \
        ! udpsink host=192.168.10.137 port=5603 sync=true

The video stream works perfectly well with the following minimal gstreamer pipeline on the ground control raspberry pi:

gst-launch-1.0 udpsrc port=5603 ! "application/x-rtp, payload=96" ! rtph264depay ! avdec_h264 ! queue ! autovideosink

However, the video stream in the QGC preview is in slow motion / has high latency . This depends on the framerate in the video-transmitting rasp pi.

Any help would be much appreciated.

NOTE: Here is my docker build script, in case it is of use to anybody. I chose debian bullseye for the base image as it has gstreamer 1.18 in its native packages - I simply had to upgrade CMake to support the QGC compile.

# Install dependencies. This lets us skip this stage for debugging.
FROM debian:bullseye

# Set a few env vars for the build stage
ENV DEBIAN_FRONTEND noninteractive
ENV DISPLAY :99

# Install QGC dependencies
COPY tools/setup/install-dependencies-debian.sh /tmp/qt/
RUN /tmp/qt/install-dependencies-debian.sh

# Set the locale to UTF-8. This is needed for the QGC build.
RUN sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen \
      && locale-gen en_US.UTF-8 \
      && dpkg-reconfigure locales \
      && update-locale LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 
ENV LANG=en_US.UTF-8

# Create a non-root user (source https://code.visualstudio.com/remote/advancedcontainers/add-nonroot-user)
# We need to run QGC as non root user, otherwise it throws an error.
RUN apt-get install -y sudo
ARG USERNAME=pi
ARG USER_UID=1000
ARG USER_GID=$USER_UID
RUN groupadd --gid $USER_GID $USERNAME \
    && useradd --uid $USER_UID --gid $USER_GID -m $USERNAME \
    # Add sudo support.
    && echo $USERNAME ALL=\(root\) NOPASSWD:ALL > /etc/sudoers.d/$USERNAME \
    && chmod 0440 /etc/sudoers.d/$USERNAME

ARG QT_VERSION=6.7.2
ENV QT_PATH /home/pi/Qt
ENV QT_DESKTOP $QT_PATH/${QT_VERSION}/gcc_arm64
ENV PATH /usr/lib/ccache:$QT_DESKTOP/bin:$PATH  

# For the arm build, we need to copy over a pre-installed version of Qt.
# I use 6.7.2 as that has been proven to work in the past, and an arm binary is available through
# the Qt online installer.
# When running the docker build, use the arg `--build-context qt-install-dir=/home/pi/Qt`
COPY --from=qt-install-dir . $QT_PATH

# Run the QGC-provided qt install script. Note that I have modified this
# to use a pre-installed Qt, instead of using aqt to download Qt.
# I still use the old scripts as it sets all the right environment variables for us.
COPY tools/setup/install-qt-debian.sh /home/pi

RUN echo $PATH
RUN /home/pi/install-qt-debian.sh

#install latest cmake
# Note for developers: To prepare this step, download the cmake install .sh file
# by running curl https://cmake.org/files/v3.25/cmake-3.25.1-linux-aarch64.sh --output ~/custom_cmake/cmake-3.25.1-linux-aarch64.sh
# During the docker build, we need to add the arg `--build-context custom_cmake=/home/pi/custom_cmake`
COPY --from=custom_cmake cmake-3.25.1-linux-aarch64.sh /home/pi/cmake-3.25.1-linux-aarch64.sh
RUN mkdir /home/pi/cmake
RUN sh /home/pi/cmake-3.25.1-linux-aarch64.sh --prefix=/home/pi/cmake --skip-license
RUN sudo ln -s /home/pi/cmake/bin/cmake /usr/local/bin/cmake

# Copy over only the necessary parts of QGroundControl for the build
COPY . /home/pi/project/source/

# IDK what this does, but it is in the QGC dockerfile.
RUN git config --global --add safe.directory /home/pi/project/source

# Build QGroundControl
RUN cmake -S /home/pi/project/source -DCMAKE_PREFIX_PATH="/home/pi/Qt/6.7.2/gcc_arm64/lib/cmake/Qt6/" -B /home/pi/project/build -G Ninja -DCMAKE_BUILD_TYPE=Release 
RUN cmake --build /home/pi/project/build --target all --config Release ;

# Install some extra xcb dependencies. Not sure if these are required, or whether you just need to run xhost +local: on the pi.
RUN apt-get update && apt-get install -y \
    libxcb-cursor0 \
    libxcb-xinerama0 \
    libxcb-randr0 \
    libxcb-xfixes0 \
    libxcb-shape0 \
    libxcb-sync-dev \
    libxcb-keysyms1 \
    libxcb-icccm4

[DronecodeBot]

This issue has been mentioned on Discussion Forum for PX4, Pixhawk, QGroundControl, MAVSDK, MAVLink. There might be relevant details there:

https://discuss.px4.io/t/qgc-locally-compiled-with-qt6-running-on-rpi5-8g/42035/2

[HTRamsey]

For the master branch I would recommend GStreamer 1.22

[DronecodeBot]

This issue has been mentioned on Discussion Forum for PX4, Pixhawk, QGroundControl, MAVSDK, MAVLink. There might be relevant details there:

https://discuss.px4.io/t/ideostreaming-delay-issues-on-raspberry-pi-build-of-qgc/42850/1

[jack-obrien]

Thank you for your prompt response! I will give that a try soon if I can.

In all fairness, I have been running my build off of a single master branch commit from about 6 months ago, just for consistency. But given your advice I will consider pulling the new changes down.