Search code examples
infinibandrdmamellanoxofed

Error using verbs Memory Windows (ibv_alloc_mw)

I am trying to use memory windows and I am getting EPERM (errno=1) when calling ibv_alloc_mw (with both types of MWs).

I have mellanox ConnectX-3 cards and the following OFED:

ofed_info | head -n 1
MLNX_OFED_LINUX-3.2-2.0.0.0 (OFED-3.2-2.0.0):

It seems my card should support this capability:

ibv_devinfo -v | grep EXP_MW
EXP_MW_TYPE_2B
EXP_MW_TYPE_2B

When running rping in the Mellanox OFED I get the same error:

librdmacm-1.1.0mlnx/examples> ./rping -s &
librdmacm-1.1.0mlnx/examples> ./rping -c -a 10.10.49.87 -w 1
rdma_buf alloc_mw failed
rping_setup_buffers failed: -12

librdmacm-1.1.0mlnx/examples> ./rping -c -a 10.10.49.87 -w 2
rdma_buf alloc_mw failed
rping_setup_buffers failed: -12

Is ibv_alloc_mw even supported by Mellanox?

EDIT:

I got access to a ConnectX-4 card and got the same error. I can also see that there is an error being logged whenever I try to run rping with the memory window option:

dmesg | tail -n 2
[9349257.246854] mlx4_core 0000:02:00.0: vhcr command 0xd slave:0in_param 0xe26b4000 in_mod=0x110 op_mod=0x0 failed with error:0, status -1
[9349257.246856] mlx4_core 0000:02:00.0: SW2HW_MPT failed (-1)
Solution

  • Upon further exploration it seems this is due to having SR-IOV enabled in the nodes I'm using. Disabling SR-IOV makes this work.