Search code examples
dpdk

How to run send and receive traffic over 2 instances of dpdk-testpmd running on the same host?

Note: I'm new to network and dpdk, so there might be some misunderstanding in fundamental concepts...

I want to run 2 instances of dpdk-testpmd on the same host to send and receive traffic over separate NIC.

Configuration:

NIC:

  • PMD: MLX5
  • version: 5.0-1.0.0.0
  • firmware-version: 16.26.1040 (MT_0000000011)
  • NUMA-Socket: same
  • PCIe: 0000:3b:00.0, 0000:3b:00.1

Update

  • DPDK Version: 20.11.1
  • Hugepage Total: 32768
  • Hugepage Free: 32768

TESTPMD Logs:

Logical Core 12 (socket 0) forwards packets on 6 streams:
  RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00

Questions:

  1. How to set the MAC address of instance 2's port in instance 1?
  2. What's the meaning of RX P=0/Q=0?
  3. Is the NIC will receive packets from it's RXQ 0(a ring buffer in memory?)
  4. Is the NIC will put packets(just got from RXQ) to it's TXQ 0?
  5. What will happen next?
  6. Where is the packet's destination?
  7. How to set/check it?

Hope for your help. Thanks in advance.

Solution

  • Note: I highly recommend reading about dpdk testpmd as it covered all your questions in detail. As per the StackOverflow guideline multiple sub-questions make answering difficult. Please use well-formatted and formed question for better reach and answers.

    @Alexcured since you have mentioned you know how to run 2 separate instances of dpdk-testpmd. I will only recommend heavily reading up on dpdk-testpmd URL which has answers to most of your questions too.

    The assumption is made, both the PCIe NIC are working properly and interconnect between 2 are tested with either arping or ping (Kernel Driver). After binding both PCIe devices to DPDK supported drivers, one should use options for DPDK 20.11.1 such as

    • use file-prefix option as unique names
    • set socket-memory to fetch memory from the desired NUMA-SOCKET
    • set socket-limit to prevent ballooning of huge page mmap
    • use w|b option to whitelist|blacklist PCIe devices (0000:3b:00.0 and 0000:3b:00.1)
    • Since it is separate physical devices ensure there physical cable connection between 2 PCIe ports.

    [Q.1] How to set the MAC address of instance 2's port in instance 1?

    [A.1] in DPDK instance-1 & instance 2, execute `show port 0 mac` to show current port MAC address. Note down the MAC address correctly. To set the MAC address of peer use command `set eth-peer (port_id) (peer_addr)`.
In non interactive mode this can be done from cmdline too.

    [Q.2] What's the meaning of RX P=0/Q=0?

    [A.2] P stands for `port`, and Q stands for `queue`. So P=0/Q=0 means `port 0/queue 0`

    [Q.3] Is the NIC will receive packets from it's RXQ 0(a ring buffer in memory?)

    [A.3] NIC can receive from port 0 - rxq -0; provided there is no offload|RSS|Rules set (since you have not shared the cmdline I will assume there is none). 
NIC HW RQ is mmaped to reflect HEAD-TAIL pointer to allow PMD to trigger DMA copy to mempool (huge page backed area). This is different from DPDK lib_rte_ring buffer area

    [Q.4] Is the NIC will put packets(just got from RXQ) to it's TXQ 0?

    [A.4] if the option `start tx-first` is not used, as per the current understanding there will not be any packets TX from the NIC (for Intel FVL LLDP packets will be send from NIC HW unless disabled).
The default configuration in looped mode, and with 1 DPDK port RX of the Port will be always sent out as TX.

    [Q.5] What will happen next?

    [A.5] Packet will be send out with modified eth-peer mac address

    [Q.6] Where is the packet's destination?

    [A.6] Ethernet Packet has the format of first 6 bytes as destination MAC address. With `set peer-address` done the MAC address as the destination will be set as value set.

    [Q.7] How to set/check it?

    [A.7] use option `set verbose 2`. This will show the RX packet details received from the port