Userspace DMA I/O

This library should simplify the use of the Xilinx AXI DMA controller when used in S2MM (Stream to Memory-Mapped) mode.

A typical use case would be a DAQ (Data Acquisition) system, where ADC is generating data and the DMA is responsible to store the data in the memory.

A configuration required by this library is presented in the block diagram below.

Different operation modes

Access interface

This library supports two different access modes:

• over PCIe with xdma driver
• from ARM over the Userspace I/O

Both methods are somehow similar, but are also different in a couple of details.

Memory mapping

xdma provides only one memory mapping for the entire register space, while uio provides a separate file for each IP.

Consequently, an address of the IP needs to be provided to the xdma handler, as this information is not made available.

Interrupts

Interrupts can be associated with an individual uio files, but not with the xdma memory mapped file; xdma instead provides separate files (with the name eventN).

Data memory access

There are also two way to provide a memory for the DMA data, i.e. the memory where the DMA is writing to:

• u-dma-buf, which provides a method to allocate a chunk of memory from the kernel
• FPGA memory, where the memory is connected directly to the FPGA (e.g. DDR4 memory connected directly to Programmable Logic (PL) on DAMC-FMC2ZUP)

Python binding

Please keep in mind that on some systems (e.g. DAMC-FMC2ZUP and DAMC-FMC1Z7IO) the libudmaio and the pyudmaio Python binding can be already installed. The steps descibed in the chapter below are only needed for the development of the library itself.

Installation

The Python binding pyudmaio uses setuptools to build/install and requires pip 21 or newer.

It also assumes the libudmaio header files and shared library to be installed system-wide. Make sure to make install the C++ library before trying to install the Python binding.

To install the Python library, in-tree build must be used:

pip install --use-feature=in-tree-build .

or (to install the library in editable mode:

pip3 install --use-feature=in-tree-build -e .

.pyi update

To update the .pyi stubs after changing the binding:

# (re-)generate docstrings
 python3 -m pybind11_mkdoc -I ./inc -o pyudmaio/src/docstrings.hpp \
  inc/udmaio/UioConfig.hpp \
  inc/udmaio/UioIf.hpp \
  inc/udmaio/DmaBufferAbstract.hpp \
  inc/udmaio/FpgaMemBufferOverAxi.hpp \
  inc/udmaio/FpgaMemBufferOverXdma.hpp \
  inc/udmaio/UDmaBuf.hpp \
  inc/udmaio/UioAxiDmaIf.hpp \
  inc/udmaio/UioMemSgdma.hpp \
  pyudmaio/src/DataHandlerPython.hpp \
  inc/udmaio/FrameFormat.hpp

# build & install binding
pip3 install ./pyudmaio

# update stubs
python3 -m pybind11_stubgen pyudmaio -o pyudmaio

Usage example

Available in the folder example is a small example, which demonstrates how to use the components of this library. This example assumes that the AXI Traffic Generator is connected to the AXI4-Stream input of the DMA; this Traffic Generator IP produces a LFSR pattern which is then compared in the interrupt handler.

udev rules

Install TechLab's xdma metapackage (either as Debian package xdma-dkms from doocs.desy.de or from source) to make sure udev rules are installed to match the device file paths expected by the library.

UIO - with default parameters

# ./axi_dma_demo --mode uio --debug
[2069-12-16 19:52:38.443477] [0x0000007fbce25010] [debug]   : uio name = /dev/uio5
[2069-12-16 19:52:38.444203] [0x0000007fbce25010] [debug]   : uio name = /dev/uio4
[2069-12-16 19:52:38.444769] [0x0000007fbce25010] [debug]   : uio name = /dev/uio6
[2069-12-16 19:52:38.444988] [0x0000007fbce25010] [debug]   UDmaBuf: size      = 134217728
[2069-12-16 19:52:38.445022] [0x0000007fbce25010] [debug]   UDmaBuf: phys addr = 6fd00000
[2069-12-16 19:52:38.445264] [0x0000007fbce25010] [debug]   UioAxiDmaIf: start, start_desc = 88840000
[2069-12-16 19:52:38.445651] [0x0000007fb46251c0] [debug]   DataHandlerPrint: process data, size = 16384
[2069-12-16 19:52:38.445853] [0x0000007fb46251c0] [debug]   DataHandler: stopping thread
Counters: OK = 8192, total = 8192

UIO - long data acquisition

When the pause between packets is high enough (and when the packets are small enough), the library can operate continuously - here an example where 50000 samples were captured.

# ./axi_dma_demo --debug --mode uio --nr_pkts 50000 --pkt_pause 60000
[2069-12-16 20:21:36.560101] [0x0000007fae16d010] [debug]   : uio name = /dev/uio5
[2069-12-16 20:21:36.560779] [0x0000007fae16d010] [debug]   : uio name = /dev/uio4
[2069-12-16 20:21:36.561316] [0x0000007fae16d010] [debug]   : uio name = /dev/uio6
[2069-12-16 20:21:36.561505] [0x0000007fae16d010] [debug]   UDmaBuf: size      = 134217728
[2069-12-16 20:21:36.561529] [0x0000007fae16d010] [debug]   UDmaBuf: phys addr = 6fd00000
[2069-12-16 20:21:36.561788] [0x0000007fae16d010] [debug]   UioAxiDmaIf: start, start_desc = 88840000
[2069-12-16 20:21:36.562072] [0x0000007fa596d1c0] [debug]   DataHandlerPrint: process data, size = 16384
[2069-12-16 20:21:36.562366] [0x0000007fa596d1c0] [debug]   DataHandlerPrint: process data, size = 16384
[2069-12-16 20:21:36.562650] [0x0000007fa596d1c0] [debug]   DataHandlerPrint: process data, size = 16384
[2069-12-16 20:21:36.562931] [0x0000007fa596d1c0] [debug]   DataHandlerPrint: process data, size = 16384
<...>
[2069-12-16 20:21:48.766523] [0x0000007fa596d1c0] [debug]   DataHandlerPrint: process data, size = 16384
[2069-12-16 20:21:48.766794] [0x0000007fa596d1c0] [debug]   DataHandlerPrint: process data, size = 16384
[2069-12-16 20:22:37.585985] [0x0000007fa596d1c0] [debug]   DataHandler: stopping thread
Counters: OK = 409600000, total = 40960000

XDMA

$ ./axi_dma_demo --mode xdma --debug
[2021-03-01 13:49:31.941347] [0x00007f6b7e56c740] [debug]   : uio name = /dev/xdma/card0/user
[2021-03-01 13:49:31.941421] [0x00007f6b7e56c740] [debug]   : uio name = /dev/xdma/card0/user
[2021-03-01 13:49:31.941451] [0x00007f6b7e56c740] [debug]   : uio name = /dev/xdma/card0/user
[2021-03-01 13:49:31.941601] [0x00007f6b7e56c740] [debug]   UioAxiDmaIf: start, start_desc = 88920000
[2021-03-01 13:49:31.942309] [0x00007f6b7e568700] [debug]   DataHandlerPrint: process data, size = 16384
[2021-03-01 13:49:31.942382] [0x00007f6b7e568700] [debug]   DataHandler: stopping thread
Counters: OK = 8192, total = 8192

Selecting the target hardware

The demo application includes hardware-related constants such as bus width and IP core addresses. To make it work, the right target hardware must be selected. The demo application currently supports DAMC-FMC2ZUP and DAMC-FMC1Z7IO.

Hardware selection for the C++ demo

The hardware is selected at build time using a CMake parameter.

cmake -DTARGET_HW=ZUP
cmake -DTARGET_HW=Z7IO

Hardware selection for the Python demo

The hardware is selected at runtime using a command line parameter.

./axi_dma_demo.py -H zup
./axi_dma_demo.py -H z7io

Source code documentation

Local build

• To create the Doxygen documentation for the library, run doxygen in the source tree root.
• To create the Doxygen documentation for the C++ example, run doxygen in the example directory.
• The Doxygen documentation is created in docs/libudmaio and docs/example, respectively.

GitHub pages

The Doxygen source code documentation of the master branch is also available on GitHub pages.