Drop in UDP Speed with USB Ethernet Adapter with DomU Kernel > 5.15

[sidhussmann]

Qubes OS release

• Qubes Release 4.2.3 (R4.2)

Brief summary

For my embedded development projects, my target devices usually boot via TFTP from my computer. Since my current laptop (Lenovo X1 Carbon 10th gen) does not have a physical ethernet port, I am forced to use a USB Ethernet Adapter. For this, I attach the USB Ethernet Adapter to my dev-embedded Qube. This worked fine with the 5.15.103-1.qubes.fc32.x86_64 kernel with TFTP speed up to 1.2 MiB/s. However, using any more recent kernel (>6.1) the TFTP transfer speed drops down to a painfully slow 45 KiB/s. My workaround until now was to keep the kernel for that Qube at version 5.15, which is very undesirable.

Steps to reproduce

• Set Kernel version on dev-embedded from 5.15.103-1.qubes.fc32.x86_64 to 6.6.63-1.qubes.fc37.x86_64
• Attach USB Ethernet Adapter to dev-embedded
• Setup TFTP/DHCP server in dev-embedded to enable network boot for embedded target (u-boot)
• Power-on the embedded target

Expected behavior

• The TFTP speed is in a similar range (~1.2 MiB/s) as with the 5.15 kernel

Actual behavior

• The TFTP speed drops to 45 KiB/s.

Further Information

I have been investigating the problem for a while now, and here are my findings using a 6.6 kernel:

Scheduling Weirdness

When I run stress -c 10 during the TFTP transfer, the speed significantly increases to ~500 KiB/s.

Power Saving Knobs

Tweaking the power saving options using TLP or USB autosuspend did impact the transfer speed.

Diagnostics

Neither the kernel log nor usbview revealed any obvious information indicating the reason for the slow speed.

USB Ethernet Adapters

I have tried the following ethernet adapters without any significant difference:

• ASIX 1G USB NIC: 0b95:1790 ASIX Electronics Corp. AX88179 Gigabit Ethernet
• Realtek 1G USB NIC 0bda:8153 Realtek Semiconductor Corp. RTL8153 Gigabit Ethernet Adapter
• Realtek 2.5G USB NIC: 0bda:8156 Realtek Semiconductor Corp. USB 10/100/1G/2.5G LAN

More Recent Kernels

Changing the DomU kernel to the following versions did not have any significant effect:

• 6.6.63
• 6.10.10
• 6.11.10

Transfer Speed in sys-usb

Setting up a tftp/dhcp server in sys-usb, the TFTP transfer speed is at a blazing fast 1.4 MiB/s.

I understand I can keep the USB Ethernet Adapter within sys-usb and route the traffic to my dev-embedded Qube. However, my networking requirements are a bit more complex than just forwarding TFTP packets:

• Forwarding TFTP/HTTP(S) boot to dev-embedded
• Accessing the internet from dev-embedded Qube via WiFi of sys-net
• Accessing the internet from my embedded target via USB Ethernet to WiFi of sys-net
• Accessing the embedded target's SSH server from dev-embedded Qube

I implemented the above requirements by attaching the USB Adapter to dev-embedded.

Before going down the rabbit hole of keeping the USB Ethernet Adapter attached to sys-usb and setting up this rather complex network topology on sys-usb, sys-firewall, dev-embedded and sys-net, I'm wondering if anyone has an idea what could be the root problem or which other knobs are there to tweak.

Doing a PCI passthrough of a USB host controller is not an option as the number of USB ports is already limited, and my two USB-C ports are on the same controller.