xdpoffload attach failed: Invalid argument

When I try to attach a BPF program in XDP offload mode, I get Invalid argument. I get the same error if attach through code or by using bpftool. Here's how I'm attaching using netlink:

err = netlink.LinkSetXdpFdWithFlags(link, objects.CollectIpsProg.FD(), 8)

And from using bpftool:

# bpftool prog loadall collect_ips.o /sys/fs/bpf/collect_ips type xdp
# bpftool net attach xdpoffload id 106 dev public
Error: interface xdpoffload attach failed: Invalid argument

I don't have any issues loading the program in driver mode, where 4 is passed to LinkSetXdpFdWithFlags.

My NIC, Mellanox MT28800 Family [ConnectX-5 Ex], should support HW offload.

My main XDP program makes calls to two different tail programs. I use BPF_MAP_TYPE_RINGBUF, BPF_MAP_TYPE_PROG_ARRAY, and BPF_MAP_TYPE_ARRAY.

Solution

Mellanox cards support some hardware offload (e.g., flow control rules), but not the offload of BPF programs as far as I know. The only Ethernet adapters out there that support BPF offloading are Netronome's cards.

One way to check this is to grep for the XDP_SETUP_PROG_HW BPF netdev command in the Linux source code:

$ git grep XDP_SETUP_PROG_HW
drivers/net/ethernet/netronome/nfp/nfp_net_common.c:    case XDP_SETUP_PROG_HW:
drivers/net/netdevsim/bpf.c:    if (bpf->command == XDP_SETUP_PROG_HW && !ns->bpf_xdpoffload_accept) {
drivers/net/netdevsim/bpf.c:    if (bpf->command == XDP_SETUP_PROG_HW) {
drivers/net/netdevsim/bpf.c:    case XDP_SETUP_PROG_HW:
include/linux/netdevice.h:      XDP_SETUP_PROG_HW,
net/core/dev.c: xdp.command = mode == XDP_MODE_HW ? XDP_SETUP_PROG_HW : XDP_SETUP_PROG;

That command is used to tell the driver to offload the BPF program to the hardware, via the ndo_bpf callback function.