This is one of the three components of the OpenNIC project. The other components are:
OpenNIC driver implements a Linux kernel driver for OpenNIC shell. It supports multiple PCI-e PFs with multiple TX/RX queues in each PF, and up to two 100Gbps ports on the same card. As of version 1.0, the driver has not implemented the ethtool routines to change the hash key and the indirection table.
The driver has been tested on under Ubuntu 18.04, 20.04, and 22.04 with multiple versions of the Linux kernel.
Follow the steps below to build the driver.
- Run
maketo compile the loadable kernel moduleonic.ko. - Connect 100Gbps cables/loopback adapters to enabled ports before insert the kernel module. Currently, the driver does not detect link status change. Thus links should be ready before loading the driver.
- Run
sudo insmod onic.koto insert the kernel module. (There is an optional parameter RS_FEC_ENABLED, which can be set to either zero or one.) - Verify that no error message is printed through
dmesg, and new devices show up inifconfigoutput.
The driver registers a net device for each PF it probed. Net devices are registered with multiple queues. The number of queues depends on the number of MSI-X vectors available through the associated PF. In particular, for PF0 which acts as the master PF, the number of queues equals to the number of MSI-X vectors minus 2, one for card-level error interrupt and one for function-level user interrupt; for other PFs, it equals to the number of MSI-X vectors minus 1. Each net device has the same number of TX and RX queues.
For each FPGA card loaded with the OpenNIC shell bitstream, the driver detects the number of CMAC instances and manages the links accordingly. Only PF0 can enable/disable the links. The default bitstream, when configured with 2 PFs and 2 CMAC instances, maps PF0 to port0 and PF1 to port1.
A loopback register is accessible from BAR2 at the offset 0x8090 and 0xC090,
for port0 and port1 respectively. One could use
pcimem to read/write PCI device
registers.
To enable loopback, write 0x1 to the loopback register. For instance, to
enable loopback on port0, issue the following command.
sudo ./pcimem /sys/devices/pci0000:d7/0000:d7:00.0/0000:d8:00.0/resource2 0x8090 w 0x1
After that, packets received by CMAC0 will be looped back to the host. Write
0x0 to disable loopback.
Here is a simple scenario to test the loopback mode. Assume the interface name
is enp216s0f0 and the IP address is 192.168.1.10.
-
Run
tcpdumpto capture packets on the interface.sudo tcpdump -i enp216s0f0 -xx -
Run
ping 192.168.1.10. -
Observe that packets captured by
tcpdumpare always duplicated.
ethtool is a Linux utility and used to control and read status of
various MAC parameters. Also, this tool is used to obtain various counter registers
(such as total good packets etc.) from the MAC.
If the tool is not found, install it from distro.
$ sudo apt install ethtool
List all options that the tool can support
$ ethtool -h
List active interfaces, activate required interface Note: assume interface name is xyz01, IP address is 192.168.1.1
$ ifconfig -a
$ ifconfig xyz01 192.168.1.1 up
Use ethtool interface to see the status Note: assume interface name is xyz01
$ ethtool xyz01
Show driver information Note: assume interface name is xyz01
$ ethtool -i xyz01
Show adapter statistics Note: assume interface name is xyz01
$ ethtool -S xyz01
To install lm-sensors framework:
$ sudo apt install lm-sensors
This installs 'sensors' application here: /usr/bin/sensors
To enable lm-sensors framework support in open-nic
a. In the file "onic_main.c", enable macro "CMS_SUPPORT"
b. build the open-nic driver as explained above
To test LM-SENSORS support in the open-nic
Note: CMS IP in the design need to be added
a, Load the kernel driver as explained above
b. Run the sensors application, to see the data
The output looks as below:
$ sensors
sn1000-onic-isa-0000
Adapter: ISA adapter
12V PEX: +12.22 V (max = +12.22 V, avg = +12.21 V)
12V AUX: +12.26 V (max = +12.26 V, avg = +12.25 V)
3V3 PEX: +3.26 V (max = +3.26 V, avg = +3.26 V)
1V8 TOP: +1.80 V (max = +1.80 V, avg = +1.80 V)
VCC INT: +0.85 V (max = +0.85 V, avg = +0.85 V)
VCC 3V3: +3.27 V (max = +3.27 V, avg = +3.27 V)
PCB TOP FRONT: +44.0°C (highest = +45.0°C)
PCB TOP REAR: +47.0°C (highest = +48.0°C)
FPGA TEMP: +58.0°C (highest = +59.0°C)
QSPF 0: +0.0°C (highest = +0.0°C)
QSPF 1: +0.0°C (highest = +0.0°C)
POWER: 38.66 W (avg = 40.42 W)
12V PEX Current: +2.07 A (max = +2.15 A, avg = +2.08 A)
12V AUX Current: +0.76 A (max = +0.81 A, avg = +0.75 A)
VCC INT Current: +10.00 A (max = +10.60 A, avg = +10.00 A)
3V3 PEX Current: +1.24 A (max = +1.29 A, avg = +1.24 A)
... ... ...
sensor output for other devices
... ... ...
... ... ...
It has been found that in some cases, DHCP clients may cause kernel panic after
inserting the kernel module. A message similar as below show up in dmesg.
[ 224.835445] BUG: unable to handle kernel paging request at ffff9d7f45effa1f
Assigning a static network address seems to solve the issue in most cases. Add
the following lines into /etc/network/interfaces with the correct interface
name, IP address.
```
auto IF_NAME
iface IF_NAME inet static
address IP_ADDRESS
```
An alternative is to uninstall DHCP. This can be done by killing any running processes using DHCP with
ps -eF | grep dhclient, and then to disable DHCP.
This seems to be related to the DHCP issue mentioned above in "Static IP Address". The recommendation is to disable DHCP.
© Copyright 2020 – 2021 Xilinx, Inc. All rights reserved.
This file contains confidential and proprietary information of Xilinx, Inc. and is protected under U.S. and international copyright and other intellectual property laws.
DISCLAIMER
This disclaimer is not a license and does not grant any rights to the materials distributed herewith. Except as otherwise provided in a valid license issued to you by Xilinx, and to the maximum extent permitted by applicable law: (1) THESE MATERIALS ARE MADE AVAILABLE "AS IS" AND WITH ALL FAULTS, AND XILINX HEREBY DISCLAIMS ALL WARRANTIES AND CONDITIONS, EXPRESS, IMPLIED, OR STATUTORY, INCLUDING BUT NOT LIMITED TO WARRANTIES OF MERCHANTABILITY, NONINFRINGEMENT, OR FITNESS FOR ANY PARTICULAR PURPOSE; and (2) Xilinx shall not be liable (whether in contract or tort, including negligence, or under any other theory of liability) for any loss or damage of any kind or nature related to, arising under or in connection with these materials, including for any direct, or any indirect, special, incidental, or consequential loss or damage (including loss of data, profits, goodwill, or any type of loss or damage suffered as a result of any action brought by a third party) even if such damage or loss was reasonably foreseeable or Xilinx had been advised of the possibility of the same.
CRITICAL APPLICATIONS
Xilinx products are not designed or intended to be fail-safe, or for use in any application requiring failsafe performance, such as life-support or safety devices or systems, Class III medical devices, nuclear facilities, applications related to the deployment of airbags, or any other applications that could lead to death, personal injury, or severe property or environmental damage (individually and collectively, "Critical Applications"). Customer assumes the sole risk and liability of any use of Xilinx products in Critical Applications, subject only to applicable laws and regulations governing limitations on product liability.
THIS COPYRIGHT NOTICE AND DISCLAIMER MUST BE RETAINED AS PART OF THIS FILE AT ALL TIMES.