I am trying to create a QueuePair with ibv_create_qp() and I have to describe the size of the Queue Pair by setting the fields of the struct ibv_qp_cap and providing it to the create function. My issue is with the max_send_wr field which corresponds to the maximum number of outstanding Work Requests that can be posted to the Send Queue of that specific Queue Pair.
According to various web resources [1,2], you are supposed to set this specific field somewhere in the range of [0 ... dev_cap.max_qp_wr] with max_qp_wr being the maximum number of outstanding work requests on any Send or Receive Queue supported by the RDMA device (information you can obtain by querying your device with ibv_devinfo -v). Whilst trying to create the QueuePair, when I am setting the field to the maximum possible value (max_qp_wr which for my device is 32K), the ibv_create_qp() function fails with an ENOMEM error. By means of trial and error, I have found out that it works fine if I reduce it to 8K.
Thus, I am interested in knowing if there a deterministic way of calculating the max_send_wr value for an RDMA device without resorting to trial and error? Furthermore, according to some [1], for a specific RDMA device, there may be other considerations for the Send or Receive Queue that prevents a QP to be created with the maximum values. What are could those RDMA device-specific considerations be and how do they affect the calculation of the max value?
[1] https://www.rdmamojo.com/2012/12/21/ibv_create_qp/
[2] https://linux-rdma.vger.kernel.narkive.com/rkR0gUjT/rdma-create-qp-and-max-send-wr
I'm not sure there's a deterministic and portable way to answer your question. In some devices, the device pose a limit on the work queue size, rather than the number of entries (e.g., Mellanox devices), so depending on the other parameters such as the maximum inline size and scatter-gather entries, the actual number of send work requests may.