Search code examples
infinibandrdma

RDMA transfer without mapping memory

I have supplementary processes which should exchange files over IB network. Then the files will be used by other processes.

The workflow is following:

  1. Create files in /dev/shm
  2. Resize files appropriately
  3. Map files in process' VM
  4. Register mmaped area with ibv_reg_mr
  5. Initiate RDMA operation for data transfer

It turned out that the bottleneck in my scheme is ib_reg_mr (I measured separately registering 3Gb memory takes 1.78 seconds). It seems that it triggers mapping of the memory region to the process's address space. Unfortunately, this operation is not needed, because the receiving process does not use this memory. The memory should be mapped and used by other processes later.

So I thought it would be wise to circumvent altering page tables for the receiving process, but I was not able to find if it is possible at all.

Could you advise me, if there is a way to initiate a data transfer on a memory region without mapping it to process' address space?

Solution

  • As far as I know there is no way to register a memory region without having it mapped to the process page tables. During registration the pages are pinned so that the mapping doesn't change while the device is accessing the pages, and this requires mapping them to the process. With on-demand-paging, you could delay the mapping of the pages into the process until the HCA actually uses them, but eventually they will be mapped.