IOb-eth

IObundle's ethernet core.

This core implements raw socket ethernet communication. This corresponds to the data link layer (2) of the OSI Model.

This core is driver-compatible with the ethmac core, as it contains a similar SWreg interface.

This peripheral can be used as a verification tool for the OpenCryptoTester project.

Integrate in SoC

• Check out IOb-SoC-SUT

Usage

The main class that describes this core is located in the iob_eth.py Python module. It contains a set of methods useful to set up and instantiate this core.

The following steps describe the process of creating an Ethernet peripheral in an IOb-SoC-based system:

1. Import the iob_eth class
2. Add the iob_eth class to the submodules list. This will copy the required sources of this module to the build directory.
3. Run the iob_eth(...) constructor to create a Verilog instance of the Ethernet peripheral.
4. To use this core as a peripheral of an IOb-SoC-based system:
5. Add the created instance to the peripherals list of the IOb-SoC-based system.
6. Use the _setup_portmap() method of IOb-SoC to map IOs of the Ethernet peripheral.
7. Write the firmware to run in the system, including the iob-eth.h C header, and use its driver functions to control this core.
8. Set the RMAC_ADDR and IOB_CONSOLE_PYTHON_ENV environment variables, as described here.

Create a virtual network interface

For simulation, we can use almost any network interface to connect the console script to the simulator's tesbench (via the eth2file script). You can select the network interface to use with the ETH_IF environment variable.

For debugging purposes, we can use a virtual network interface dedicated to the project. Use the following command to create a virtual network interface named 'eth10':

make virtual-network-if ETH_IF=eth10

We can later remove it by running the following command:

make remove-virtual-network-if ETH_IF=eth10

Example configuration

The iob_soc_sut.py script of the IOb-SoC-SUT system, uses the following lines of code to instantiate an Ethernet peripheral with the instance name ETH0:

# Import the iob_eth class
from iob_eth import iob_eth

# Class of the SUT system
class iob_soc_sut(iob_soc):
  ...
  @classmethod
  def _create_submodules_list(cls):
      """Create submodules list with dependencies of this module"""
      super()._create_submodules_list(
          [
              iob_eth,
              ...
          ]
      )
  # Method that runs the setup process of the SUT system
  @classmethod
  def _specific_setup(cls):
    ...
    # Create a Verilog instance of this module, named 'ETH0', and add it to the peripherals list of the system.
    cls.peripherals.append(
        iob_eth(
            "ETH0",
            "Ethernet interface",
            # These parameters configure the core's memory interface to match the system's memory.
            parameters={
                "AXI_ID_W": "AXI_ID_W",
                "AXI_LEN_W": "AXI_LEN_W",
                "AXI_ADDR_W": "AXI_ADDR_W",
                "AXI_DATA_W": "AXI_DATA_W",
                "MEM_ADDR_OFFSET": "MEM_ADDR_OFFSET",
            },
        )
    )
  ...
  # SUT system method to map IOs of peripherals
  @classmethod
  def _setup_portmap(cls):
      super()._setup_portmap()
      cls.peripheral_portmap += [
          ...
          # ETH0 IO --- Connect IOs of the Ethernet core
          # interrupt - connect to interrupt signal
          (
              {
                  "corename": "ETH0",
                  "if_name": "general",
                  "port": "inta_o",
                  "bits": [],
              },
              {
                  "corename": "internal",
                  "if_name": "ETH0",
                  "port": "",
                  "bits": [],
              },
          ),
          # phy - connect to external interface
          (
              {
                  "corename": "ETH0",
                  "if_name": "phy",
                  "port": "MTxClk",
                  "bits": [],
              },
              {
                  "corename": "external",
                  "if_name": "ETH0",
                  "port": "",
                  "bits": [],
              },
          ),
          (
              {
                  "corename": "ETH0",
                  "if_name": "phy",
                  "port": "MTxD",
                  "bits": [],
              },
              {
                  "corename": "external",
                  "if_name": "ETH0",
                  "port": "",
                  "bits": [],
              },
          ),
          (
              {
                  "corename": "ETH0",
                  "if_name": "phy",
                  "port": "MTxEn",
                  "bits": [],
              },
              {
                  "corename": "external",
                  "if_name": "ETH0",
                  "port": "",
                  "bits": [],
              },
          ),
          (
              {
                  "corename": "ETH0",
                  "if_name": "phy",
                  "port": "MTxErr",
                  "bits": [],
              },
              {
                  "corename": "external",
                  "if_name": "ETH0",
                  "port": "",
                  "bits": [],
              },
          ),
          (
              {
                  "corename": "ETH0",
                  "if_name": "phy",
                  "port": "MRxClk",
                  "bits": [],
              },
              {
                  "corename": "external",
                  "if_name": "ETH0",
                  "port": "",
                  "bits": [],
              },
          ),
          (
              {
                  "corename": "ETH0",
                  "if_name": "phy",
                  "port": "MRxDv",
                  "bits": [],
              },
              {
                  "corename": "external",
                  "if_name": "ETH0",
                  "port": "",
                  "bits": [],
              },
          ),
          (
              {
                  "corename": "ETH0",
                  "if_name": "phy",
                  "port": "MRxD",
                  "bits": [],
              },
              {
                  "corename": "external",
                  "if_name": "ETH0",
                  "port": "",
                  "bits": [],
              },
          ),
          (
              {
                  "corename": "ETH0",
                  "if_name": "phy",
                  "port": "MRxErr",
                  "bits": [],
              },
              {
                  "corename": "external",
                  "if_name": "ETH0",
                  "port": "",
                  "bits": [],
              },
          ),
          (
              {
                  "corename": "ETH0",
                  "if_name": "phy",
                  "port": "MColl",
                  "bits": [],
              },
              {
                  "corename": "external",
                  "if_name": "ETH0",
                  "port": "",
                  "bits": [],
              },
          ),
          (
              {
                  "corename": "ETH0",
                  "if_name": "phy",
                  "port": "MCrS",
                  "bits": [],
              },
              {
                  "corename": "external",
                  "if_name": "ETH0",
                  "port": "",
                  "bits": [],
              },
          ),
          (
              {
                  "corename": "ETH0",
                  "if_name": "phy",
                  "port": "MDC",
                  "bits": [],
              },
              {
                  "corename": "external",
                  "if_name": "ETH0",
                  "port": "",
                  "bits": [],
              },
          ),
          (
              {
                  "corename": "ETH0",
                  "if_name": "phy",
                  "port": "MDIO",
                  "bits": [],
              },
              {
                  "corename": "external",
                  "if_name": "ETH0",
                  "port": "",
                  "bits": [],
              },
          ),
          (
              {
                  "corename": "ETH0",
                  "if_name": "phy",
                  "port": "phy_rstn_o",
                  "bits": [],
              },
              {
                  "corename": "external",
                  "if_name": "ETH0",
                  "port": "",
                  "bits": [],
              },
          ),
      ]
  ...
  @classmethod
  def _post_setup(cls):
    ...
        if cls.is_top_module:
            # Append ethernet variables to the `config_build.mk` file of the build directory
            append_str_config_build_mk(
                """
                # Mac address of PC network interface connected to the ethernet peripheral
                RMAC_ADDR ?=989096c0632c
                export RMAC_ADDR
                # Path to custom python interpreter with `CAP_NET_RAW` capability
                IOB_CONSOLE_PYTHON_ENV ?= /opt/pyeth3/bin/python
                # IOB_CONSOLE_PYTHON_ENV ?= submodules/ETHERNET/scripts/pyRawWrapper/pyRawWrapper
                """,
                cls.build_dir,
            )

Common Issues

Obtaining RMAC_INTERFACE value:

• on the machine that connects to the FPGA Board via ethernet run: ip a
• this command will show multiple interfaces (at least 3):
  • lookback interface (with inet 127.0.0.1)
  • interface with internet connection: (enAAAA:)
    • this interface has inet set to the machine IP
    • this is the ethernet connection to the router/internet
  • another interface named enBBBB but without inet values
    • this should be the one connected to the FPGA BOARD
• set RMAC_INTERFACE=enBBBB in your ~/.bashrc

System simulation takes a long time to initialize ethernet:

• ethernet simulation has an internal reset counter, configurable by a Verilog parameter
  • check for PHY_RST_CNT in ETHERNET/hardware/src/iob_eth.v for more details

No permissions to open raw sockets

• Running raw sockets requires elevated privileges. This is solved by configuring a dedicated python3 virtual environment where the interpreter has raw socket capabilities. Alternatively you can use the python wrapper mentioned in the python wrapper section.

No data transferred between the Host Machine and FPGA Board:

• Make sure that the interface is configured with speed = 100Mb/s and duplex = full
  • check interface status with: ethtool $RMAC_INTERFACE
    • check the speed, duplex, and Auto-negotiation fields
• you can setup the interface with:

  sudo ethtool -s $RMAC_INTERFACE speed 100 duplex full autoneg off
  

Failed to bind socket on PC-Emul

• The pc-emul drivers create a local socket file in /tmp/tmpLocalSocket, but this socket is never closed by the drivers. This behavior mirrors the embedded version, where there is no equivalent operation to close sockets.
• To remove the local socket file add the following target to pc-emul/Makefile clean target:

# pc-emul/Makefile clean target
clean:
    (...)
    make clean-eth-socket

Brief description of C interface

List of available bare-metal driver functions:

// Alternative to `eth_init_clear_cache` and `eth_init_mac`. Auto-initializes MAC addresses.
void eth_init(int base_address, void (*clear_cache_func)(void));

// Set function to clear cache
void eth_init_clear_cache( void (*clear_cache_func)(void) );

// Set Ethernet base address and MAC addresses
void eth_init_mac(int base_address, uint64_t mac_addr, uint64_t dest_mac_addr);

// Get payload size from the given buffer descriptor
unsigned short int eth_get_payload_size(unsigned int idx);

// Set payload size in the given buffer descriptor
void eth_set_payload_size(unsigned int idx, unsigned int size);

// Care when using this function directly, too small a size or too large might not work (frame does not get sent)
void eth_send_frame(char *data_to_send, unsigned int size);

/* Function name: eth_rcv_frame
 * Inputs:
 * 	- data_rcv: char array where data received will be saved
 * 	- size: number of bytes to be received
 * 	- timeout: number of cycles (approximately) in which the data should be received
 * Output: 
 * 	- Return -1 if a timeout occurs (no data received), or 0 if data is
 * 	successfully received
 */
int eth_rcv_frame(char *data_rcv, unsigned int size, int timeout);

// Set timeout for receive operations
void eth_set_receive_timeout(unsigned int timeout);

// Receive a file
unsigned int eth_rcv_file(char *data, int size);

// Send a file
unsigned int eth_send_file(char *data, int size);

// Receive a file of unknown size
unsigned int eth_rcv_variable_file(char *data);

// Send a file and send the file size during the handshake, so that the receiver can know its size
unsigned int eth_send_variable_file(char *data, int size);

// Delay until phy reset is released
void eth_wait_phy_rst();

// Print ethernet status
void eth_print_status(void);

Generate Quartus IP

Use the following command to generate the ddio_out_clkbuf IP module.

/opt/intelFPGA/20.1/nios2eds/nios2_command_shell.sh qmegawiz -silent wizard=altddio_out \
INTENDED_DEVICE_FAMILY="Cyclone V" \
INVERT_OUTPUT=OFF \
LPM_HINT=UNUSED \
LPM_TYPE=altddio_out \
WIDTH=1 \
DEVICE_FAMILY="Cyclone V" \
CBX_AUTO_BLACKBOX=ALL \
ddio_out_clkbuf.v

More info: https://cdrdv2-public.intel.com/705131/ug_intro_to_megafunctions_131-683102-705131.pdf

Python wrappper with CAP_NET_RAW

This repository includes a python wrapper located in scripts/pyRawWrapper/ directory, that can be used to provide the CAP_NET_RAW capability to the python scripts.

To build the python wrapper, run the following command (sudo password required):

make -C scripts/pyRawWrapper

We can then use that python wrapper to launch scripts that need the CAP_NET_RAW capability:

./scripts/pyRawWrapper/pyRawWrapper <python_script>

For multi-user installations, it is recommended to move the wrapper to a system-wide location and set the user/group permissions accordingly. Note that any user with execute permissions for this wrapper will be able to run any program with the CAP_NET_RAW capability, allowing the user to sniff and manipulate all network traffic.

Acknowledgement

The OpenCryptoTester project is funded through the NGI Assure Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG Communications Networks, Content and Technology under grant agreement No 957073.