Search code examples
crustipcdpdk

How to do IPC with a DPDK process involved?

My goal is to create a DPDK app which will act as a middle-man between a virtual machine manager (which in my case is an userspace process) and the NIC Hardware.

So far I have tried to do something on a smaller scale. Instead of using the VMM, I created a dummy process using C language. I managed to "bind" the dummy process to the DPDK process using named semaphores and shared memory. Basically in this little demo, the DPDK app is reading from the RX buffer and puts the content on the shared memory. Then the dummy process gets the data and prints it to stdout.

All of the DPDK support for multi process communication is targeting the specific case where both apps are using the dpdk's libraries. I am wondering if there is some sort of support for the case where one app is not using those libraries. Why? Because the VMM is written in Rust and I do not know how to add DPDK libraries in Rust.

What do you think it would be the most efficient way of communication? I was thinking if it is possible to put the mempool inside the shared memory and access the mbufs directly from the dummy process.

I am currently using dpdk 20.11 Ubuntu 20.04

Thank you!

UPDATE 1:

is your question Can I interface/interact DPDK application with non DPDK application

What I am actually struggling to find is this: how do I efficiently move data received on RX buffer to a non dpdk app?

My current approach is this: https://i.sstatic.net/67Vn9.jpg

That is the main logic loop for a dpdk app which gets data on RX buffer and sends it to the "non dpdk app".

How it is happening:

  1. Read data from RX buffer.
  2. Wait untill the non dpdk app says "I am not using the shared memory, you can write on it"
  3. Write on the shared memory (nb_rx is written instead of the whole packet just for simplicity)
  4. Signal the "non dpdk" app that now the Shared Memory is available for being read.

As one can see, it is not quite efficient and I am afraid my synchronization method will create a bottleneck.

So this makes me wonder "are there any better, by the book ways, of accomplishing this communication?

Solution

  • There are 3 ways to solve HOST to GUEST/Docker problem.

    1. Common way: Allow all physical NIC in dpdk application like SPP, OVS, VPP and DPDK priamry-secodnary to leverage virtio, vhost library, memif or shared MMAP Huge page to allow copy/zero-copy mode to VM/Docker.
    2. Complex Copy way: create a shared memory location between DPDK application on host and non-DPDK application that runs in HOST/GUEST/Docker.
    3. Mimic Zero Copy way: the non DPDK application create DMA buffer areas in shared memory at fixed location. In DPDK applciation use external Memory Buffer MBUF for physical ports. DPDK PMD (that supports external MBUF) can then DMA the packet to shared area.

    Since option 2 and 3 are not common, let me explain how you might end up developing the solution.

    Option-2:

    • develop simple non DPDK application using shared mmap area, divide the area into fixed packet size (max size). Then distribute one half for TX and half for RX.
    • Initialize the DPDK application to make use of the mmap area that is created.
    • Maintain packet access with atomic head and tail pointers
    • DPDK application after RX-burst when it receives packets will get a blank index by querying head pointer, then memcopy to packet to a specific index. Once completed DPDK application will invoke rte_mbuf_free.
    • Non-DPDK application can go in use tail pointer to get valid RX packet from shared memory.
    • Perform similar operation for TX using separate location index and head/tail poitners.

    Disadvantages:

    1. packet throughput is heavily reduced.
    2. copy packet memory utilizes CPU CYCLES
    3. complex common library to maintain index, head and tail pointers for rx and tx.
    4. memory space over-provisioned for the largest packet since traffic can not be predictable.

    option-3:

    1. Create shared mmap with posix_memalign API, multiple region of 2000Byte.
    2. Use simple Data Structure (descriptor) to hold {virtual address, physical address, length, value}
    3. Create SHM area with each index populated with the above format.
    4. initialize DPDK application to access both SHM and MMAPed area
    5. In DPDK application to populate rte_pktmbuf_pool_create_extbuf, where ext_mem represents the DMA region populated under SHM physical address (not sure if this will work as the original intent was different purpose).
    6. register the callback handler to do garbage collection for once we rx_burst the packet.
    7. In TX, there are 2 options, a) easiest way to simply copy the buffer to rte_mbuf and b) create an indirect buffer to attach rte_mbuf with external buffer, wait till NIC actually send the packet (via completion queue)

    Disadvantages of option-2:

    1. complex way fo using zero copy in RX side
    2. copy mode is easiest method to implement.
    3. Buffer management is fragile.
    4. assuming 1 thread for RX-TX, is applicable.

    Recommendation: if the intention is not to use VPP, SPP, OVS then simplest approach is use Primary-Secondary DPDK format where all the RX mbuf and TX mbuf is available between 2 process as it is mmaped in hugepage.