Search code examples
dpdk

How do I enable Rx jumbo frame offload for DPDK on Mellanox ConnectX-4?

I am trying to run a DPDK application on Ubuntu 20.04.6 LTS. I am using the following components:

  • Ubuntu 20.04.6 LTS
$ uname -a
Linux the-hostname-001 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
  • Mellanox Technologies MT27700 Family [ConnectX-4]
$ lspci -vvv | grep -i mellanox
d8:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
        Subsystem: Mellanox Technologies ConnectX-4 Stand-up dual-port 100GbE MCX416A-CCAT
d8:00.1 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
        Subsystem: Mellanox Technologies ConnectX-4 Stand-up dual-port 100GbE MCX416A-CCAT

I am trying to run a simple bridge between two DPDK ports, using the following click router config:

FromDPDKDevice(0, PROMISC true, JUMBO true, MTU 9000) -> ToDPDKDevice(1);
FromDPDKDevice(1, PROMISC true, JUMBO true, MTU 9000) -> ToDPDKDevice(0);

My interfaces are configured for MTU 9000:

$ ip link sh

4: ens50f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether <hidden mac addr> brd ff:ff:ff:ff:ff:ff
5: ens50f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether <hidden mac addr> brd ff:ff:ff:ff:ff:ff

When I run the program, I get the following output:

$ sudo ./fastclick-main/bin/click  --dpdk -a d8:00.0 -a d8:00.1 -c 0xFFF0 -n 4 --socket-mem 16000,16000 -- --file bridge.click
EAL: Detected CPU lcores: 48
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:1013) device: 0000:d8:00.0 (socket 1)
mlx5_net: Default miss action is not supported.
EAL: Probe PCI driver: mlx5_pci (15b3:1013) device: 0000:d8:00.1 (socket 1)
mlx5_net: Default miss action is not supported.

Initializing DPDK
/home/user1/bridge.click:2: While initializing ‘ToDPDKDevice@4 :: ToDPDKDevice’:
  Rx jumbo frame offload is not supported by this device!

I am wondering if there is some standard way to ensure that my ml5x device supports "RX jumbo frame offload". I realize that the error message is specific to the fastclick application, but this feels like more of a DPDK + mlx5 device issue. I'm wondering if I missed a flag during the DPDK build process or something.

I ensured that the interfaces are enabled by the system at boot with their MTU set to 9000, but I'm still getting this error message.

Solution

  • This ended up being an issue where the DPDK mlx5 driver was setting a very high value for max_rx_pktlen

    info->max_rx_pktlen = 65536;

    After changing that value to my desired MTU, I was able to get past this issue.