I implementing the Poll Mode Driver(PMD) memif(vdev=net_memif) virtual device in application. Specificly, I have to run memif as a client in secondary process.According to the Dpdk pmd multiprocess application specification, I have to initialize memif vdev at primary process and use the device at secondary process. With this information I can implement the server memif vdev which can run at secondary process.The steps that I have done to make it work like below
./primaryApp --proc-type=primary --file-prefix=pmd1 --vdev=net_memif,role=server,socket=/tmp/memif.sock, socket-abstract=no
./secondaryApp --proc-type=auto --file-prefix=pmd1
.But in opposite configuration (run memif as client at secondary process), I cannot able to run by implementing the same steps.
I have a question want to ask.
This is the configuration step for memif eth device which runs at secondary process,
if(!rte_eth_dev_get_port_by_name(m_name.c_str(),&m_portId))
std::cout << "Get net_memif eth device successfully port id : " << m_portId << "\n";
else
{
std::cout << "rte_eth_dev_get_port_by_name ERROR \n";
return false;
}
auto eth_count = rte_eth_dev_count_avail();
if(eth_count < 0)
return false;
if(!rte_eth_dev_info_get(m_portId,ðInfo)) {
std::cout << "Get eth dev info successfully\n";
std::cout << " port id : " << m_portId <<"ethInfo : " << ethInfo.driver_name << "\n";
}
else
{
std::cout << "Get eth dev info ERROR\n";
return false;
}
if(!rte_eth_dev_configure(m_portId, m_config.queue_rx_count, m_config.queue_tx_count, ðConfig)) {
std::cout << "Configure eth dev successfully\n";
}
else
{
std::cout << "Configure eth dev ERROR\n";
return false;
}
if(!rte_eth_dev_adjust_nb_rx_tx_desc(m_portId, &m_config.queue_rx_desc, &m_config.queue_tx_desc))
std::cout << "Adjust eth dev rx tx desc successfully\n";
else
{
std::cout << "Adjust eth dev rx tx desc ERROR\n";
return false;
}
for (int i = 0; i < m_config.queue_rx_count; i++) {
if(!rte_eth_rx_queue_setup(m_portId, i, m_config.queue_rx_desc, rte_eth_dev_socket_id(m_portId), ðInfo.default_rxconf, mempool))
std::cout << "Setup rx queue successfully socket id : " << rte_eth_dev_socket_id(m_portId) << std::endl;
else
{
std::cout << "Setup rx queue ERROR\n";
return false;
}
}
for (int i = 0; i < m_config.queue_tx_count; i++) {
if(!rte_eth_tx_queue_setup(m_portId, i, m_config.queue_tx_desc, rte_eth_dev_socket_id(m_portId), ðInfo.default_txconf))
std::cout << "Setup tx queue successfully socket id : " << rte_eth_dev_socket_id(m_portId) << "\n";
else
{
std::cout << "Setup tx queue ERROR\n";
return false;
}
}
then I am starting the memif eth device as fallows,
if( rte_eth_promiscuous_enable(m_portId) != 0) {
std::cout << "rte_eth_promiscuous_enable error\n";
return false;
}
else
std::cout << "rte_eth_promiscuous_enable success \n";
if(!rte_eth_dev_start(m_portId)) {
std::cout << "Eth dev is started \n";
}
else {
std::cout << "Eth dev is start ERROR \n";
return false;
}
Can anybody give me a clue about this issue ? I could not find an information such an implementation and dpdk documentations are not well documented. Any help will be appreciated. The primary process detailed output (it just run rte_eal_init()):
sudo bin/test-memif-dummy --proc-type=primary --file-prefix=pmd2 --vdev=net_memif0,id=0,role=client,socket=/tmp/memif_test.sock,socket-abstract=no --socket-mem=512 --legacy-mem --log-level=eal,8
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Detected lcore 4 as core 0 on socket 0
EAL: Detected lcore 5 as core 1 on socket 0
EAL: Detected lcore 6 as core 2 on socket 0
EAL: Detected lcore 7 as core 3 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Checking presence of .so 'librte_eal.so.21.0'
EAL: Checking presence of .so 'librte_eal.so.21'
EAL: Checking presence of .so 'librte_eal.so'
EAL: Detected static linkage of DPDK
EAL: Ask a virtual area of 0x7000 bytes
EAL: Virtual area found at 0x100000000 (size = 0x7000)
EAL: Multi-process socket /var/run/dpdk/pmd2/mp_socket
EAL: DPAA Bus not present. Skipping.
EAL: VFIO PCI modules not loaded
EAL: Bus dpaa_bus wants IOVA as 'DC'
EAL: Bus fslmc wants IOVA as 'DC'
EAL: Bus pci wants IOVA as 'DC'
EAL: Buses did not request a specific IOVA mode.
EAL: Module /sys/module/rte_kni not found! error 2 (No such file or directory)
EAL: IOMMU is not available, selecting IOVA as PA mode.
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: IOMMU type 1 (Type 1) is supported
EAL: IOMMU type 7 (sPAPR) is not supported
EAL: IOMMU type 8 (No-IOMMU) is not supported
EAL: VFIO support initialized
EAL: Ask a virtual area of 0x2e000 bytes
EAL: Virtual area found at 0x100007000 (size = 0x2e000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 1048576
EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824
EAL: Detected memory type: socket_id:0 hugepage_sz:2097152
EAL: Creating 2 segment lists: n_segs:32 socket_id:0 hugepage_sz:1073741824
EAL: Ask a virtual area of 0x1000 bytes
EAL: Virtual area found at 0x100035000 (size = 0x1000)
EAL: Memseg list allocated at socket 0, page size 0x100000kB
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x140000000 (size = 0x800000000)
EAL: VA reserved for memseg list at 0x140000000, size 800000000
EAL: Ask a virtual area of 0x1000 bytes
EAL: Virtual area found at 0x940000000 (size = 0x1000)
EAL: Memseg list allocated at socket 0, page size 0x100000kB
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x980000000 (size = 0x800000000)
EAL: VA reserved for memseg list at 0x980000000, size 800000000
EAL: Creating 4 segment lists: n_segs:8192 socket_id:0 hugepage_sz:2097152
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x1180000000 (size = 0x61000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x1180200000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x1180200000, size 400000000
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x1580200000 (size = 0x61000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x1580400000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x1580400000, size 400000000
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x1980400000 (size = 0x61000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x1980600000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x1980600000, size 400000000
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x1d80600000 (size = 0x61000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x1d80800000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x1d80800000, size 400000000
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: Hugepage /mnt/hugepages/pmd2map_1 is on socket 0
EAL: Hugepage /mnt/hugepages/pmd2map_0 is on socket 0
EAL: Requesting 1 pages of size 1024MB from socket 0
EAL: Attempting to map 1024M on socket 0
EAL: Allocated 1024M on socket 0
EAL: Added 1024M to heap on socket 0
EAL: TSC frequency is ~2000000 KHz
EAL: Main lcore 0 is ready (tid=7f12244c1c00;cpuset=[0])
EAL: lcore 1 is ready (tid=7f12234ba400;cpuset=[1])
EAL: lcore 2 is ready (tid=7f1222cb9400;cpuset=[2])
EAL: lcore 3 is ready (tid=7f12224b8400;cpuset=[3])
EAL: lcore 4 is ready (tid=7f1221cb7400;cpuset=[4])
EAL: lcore 5 is ready (tid=7f12214b6400;cpuset=[5])
EAL: lcore 6 is ready (tid=7f1220cb5400;cpuset=[6])
EAL: lcore 7 is ready (tid=7f1203fff400;cpuset=[7])
EAL: PCI device 0000:00:1f.6 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:15d8 net_e1000_em
EAL: Not managed by a supported kernel driver, skipped
EAL: No legacy callbacks, legacy socket not created
Application is started
EAL: msg: bus_vdev_mp
EAL: sendmsg: bus_vdev_mp
EAL: reply: bus_vdev_mp
EAL: msg: eal_vfio_mp_sync
EAL: reply: eal_vfio_mp_sync
The secondary process detailed output:
sudo bin/test-memif-client --proc-type=auto --file-prefix=pmd2 --vdev=net_memif0,id=0,role=client,socket=/tmp/memif_test.sock,socket-abstract=no --socket-mem=512 --legacy-mem --log-level=eal,8
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Detected lcore 4 as core 0 on socket 0
EAL: Detected lcore 5 as core 1 on socket 0
EAL: Detected lcore 6 as core 2 on socket 0
EAL: Detected lcore 7 as core 3 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Auto-detected process type: SECONDARY
EAL: Checking presence of .so 'librte_eal.so.21.0'
EAL: Checking presence of .so 'librte_eal.so.21'
EAL: Checking presence of .so 'librte_eal.so'
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/pmd2/mp_socket_77819_23c2a1740b47
EAL: DPAA Bus not present. Skipping.
EAL: VFIO PCI modules not loaded
EAL: request: bus_vdev_mp
EAL: msg: bus_vdev_mp
EAL: msg: bus_vdev_mp
EAL: Bus dpaa_bus wants IOVA as 'DC'
EAL: Bus fslmc wants IOVA as 'DC'
EAL: Bus pci wants IOVA as 'DC'
EAL: Buses did not request a specific IOVA mode.
EAL: Module /sys/module/rte_kni not found! error 2 (No such file or directory)
EAL: IOMMU is not available, selecting IOVA as PA mode.
EAL: Selected IOVA mode 'PA'
EAL: Probing VFIO support...
EAL: request: eal_vfio_mp_sync
EAL: msg: eal_vfio_mp_sync
EAL: VFIO support initialized
EAL: Ask a virtual area of 0x2e000 bytes
EAL: Virtual area found at 0x100007000 (size = 0x2e000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 1048576
EAL: Ask a virtual area of 0x1000 bytes
EAL: Virtual area found at 0x100035000 (size = 0x1000)
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x140000000 (size = 0x800000000)
EAL: VA reserved for memseg list at 0x140000000, size 800000000
EAL: Ask a virtual area of 0x1000 bytes
EAL: Virtual area found at 0x7f8e7b87b000 (size = 0x1000)
EAL: WARNING: Address Space Layout Randomization (ASLR) is enabled in the kernel.
EAL: This may cause issues with mapping memory into secondary processes
EAL: Analysing 2 files
EAL: Main lcore 0 is ready (tid=7f8e77d50c00;cpuset=[0])
EAL: lcore 1 is ready (tid=7f8e76d49400;cpuset=[1])
EAL: lcore 2 is ready (tid=7f8e76548400;cpuset=[2])
EAL: lcore 4 is ready (tid=7f8e75546400;cpuset=[4])
EAL: lcore 3 is ready (tid=7f8e75d47400;cpuset=[3])
EAL: lcore 5 is ready (tid=7f8e74d45400;cpuset=[5])
EAL: lcore 7 is ready (tid=7f8e5f7fe400;cpuset=[7])
EAL: lcore 6 is ready (tid=7f8e5ffff400;cpuset=[6])
EAL: PCI device 0000:00:1f.6 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:15d8 net_e1000_em
EAL: Not managed by a supported kernel driver, skipped
EAL: No legacy callbacks, legacy socket not created
pmd device init is called
Get net_memif eth device successfully port id : 0
Get eth dev info successfully
port id : 0ethInfo : net_memif
Configure eth dev successfully
Adjust eth dev rx tx desc successfully
Setup rx queue successfully socket id : -1
Setup tx queue successfully socket id : -1
rte_eth_promiscuous_enable success
Eth dev is started
Port 0 Link Down
link up error Resource temporarily unavailable
Port 0 Link Down
link up error Resource temporarily unavailable
Port 0 Link Down
link up error Resource temporarily unavailable
Port 0 Link Down
The third application output (running server) :
sudo bin/test-memif-server --proc-type=auto --file-prefix=pmd1 --vdev=net_memif0,id=0,role=server,socket=/tmp/memif_test.sock,socket-abstract=no --socket-mem=512 --log-level=eal,8
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Detected lcore 4 as core 0 on socket 0
EAL: Detected lcore 5 as core 1 on socket 0
EAL: Detected lcore 6 as core 2 on socket 0
EAL: Detected lcore 7 as core 3 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Auto-detected process type: PRIMARY
EAL: Checking presence of .so 'librte_eal.so.21.0'
EAL: Checking presence of .so 'librte_eal.so.21'
EAL: Checking presence of .so 'librte_eal.so'
EAL: Detected static linkage of DPDK
EAL: Ask a virtual area of 0x7000 bytes
EAL: Virtual area found at 0x100000000 (size = 0x7000)
EAL: Multi-process socket /var/run/dpdk/pmd1/mp_socket
EAL: DPAA Bus not present. Skipping.
EAL: VFIO PCI modules not loaded
EAL: Bus dpaa_bus wants IOVA as 'DC'
EAL: Bus fslmc wants IOVA as 'DC'
EAL: Bus pci wants IOVA as 'DC'
EAL: Buses did not request a specific IOVA mode.
EAL: Module /sys/module/rte_kni not found! error 2 (No such file or directory)
EAL: IOMMU is not available, selecting IOVA as PA mode.
EAL: Selected IOVA mode 'PA'
EAL: No free hugepages reported in hugepages-2048kB
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: IOMMU type 1 (Type 1) is supported
EAL: IOMMU type 7 (sPAPR) is not supported
EAL: IOMMU type 8 (No-IOMMU) is not supported
EAL: VFIO support initialized
EAL: Ask a virtual area of 0x2e000 bytes
EAL: Virtual area found at 0x100007000 (size = 0x2e000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 1048576
EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824
EAL: Detected memory type: socket_id:0 hugepage_sz:2097152
EAL: Creating 2 segment lists: n_segs:32 socket_id:0 hugepage_sz:1073741824
EAL: Ask a virtual area of 0x1000 bytes
EAL: Virtual area found at 0x100035000 (size = 0x1000)
EAL: Memseg list allocated at socket 0, page size 0x100000kB
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x140000000 (size = 0x800000000)
EAL: VA reserved for memseg list at 0x140000000, size 800000000
EAL: Ask a virtual area of 0x1000 bytes
EAL: Virtual area found at 0x940000000 (size = 0x1000)
EAL: Memseg list allocated at socket 0, page size 0x100000kB
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x980000000 (size = 0x800000000)
EAL: VA reserved for memseg list at 0x980000000, size 800000000
EAL: Creating 4 segment lists: n_segs:8192 socket_id:0 hugepage_sz:2097152
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x1180000000 (size = 0x61000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x1180200000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x1180200000, size 400000000
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x1580200000 (size = 0x61000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x1580400000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x1580400000, size 400000000
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x1980400000 (size = 0x61000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x1980600000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x1980600000, size 400000000
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x1d80600000 (size = 0x61000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x1d80800000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x1d80800000, size 400000000
EAL: Allocating 1 pages of size 1024M on socket 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: Added 1024M to heap on socket 0
EAL: TSC frequency is ~2000000 KHz
EAL: Main lcore 0 is ready (tid=7fb885decc00;cpuset=[0])
EAL: lcore 1 is ready (tid=7fb884de5400;cpuset=[1])
EAL: lcore 3 is ready (tid=7fb87f7fe400;cpuset=[3])
EAL: lcore 2 is ready (tid=7fb87ffff400;cpuset=[2])
EAL: lcore 5 is ready (tid=7fb87e7fc400;cpuset=[5])
EAL: lcore 4 is ready (tid=7fb87effd400;cpuset=[4])
EAL: lcore 7 is ready (tid=7fb87d7fa400;cpuset=[7])
EAL: lcore 6 is ready (tid=7fb87dffb400;cpuset=[6])
EAL: PCI device 0000:00:1f.6 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:15d8 net_e1000_em
EAL: Not managed by a supported kernel driver, skipped
EAL: No legacy callbacks, legacy socket not created
8191
pmd device init is called
Get net_memif eth device successfully port id : 0
Get eth dev info successfully
port id : 0ethInfo : net_memif
Configure eth dev successfully
Adjust eth dev rx tx desc successfully
Setup rx queue successfully socket id : -1
Setup tx queue successfully socket id : -1
rte_eth_promiscuous_enable success
Eth dev is started
Port 0 Link Down
Port 0 Link Down
Port 0 Link Down
Port 0 Link Up - speed 10000 Mbps - full-duplex
Application is started
Server receiving
There are some gaps in understanding DPDK Primary-Secondary process and how memif PMD works in DPDK. Let me try to explain
[Answer] based on the code snippet, memif PMD is getting used in custom applications. There is no code snippet indicating memif PMD is modified or enhanced for the requirement.
[Answer] yes, memif can be used in the secondary process. That is, to a primary process that creates a memif interface as server or client; a secondary process can be attached. From the secondary process one can allow rx_burst and tx_burst too. But the memif device instance is created by the primary to be accessed by secondary.
Can one create a new memif interface in secondary? Answer is NO
, because of the memif PMD probe code always checks process type as secondary and tries to attach to memif interface in secondary.
File: drivers/net/memif/rte_eth_memif.c
Function: rte_pmd_memif_probe
Code:
if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
eth_dev = rte_eth_dev_attach_secondary(name);
if (!eth_dev) {
MIF_LOG(ERR, "Failed to probe %s", name);
return -1;
}
eth_dev->dev_ops = &ops;
eth_dev->device = &vdev->device;
eth_dev->rx_pkt_burst = eth_memif_rx;
eth_dev->tx_pkt_burst = eth_memif_tx;
if (!rte_eal_primary_proc_alive(NULL)) {
MIF_LOG(ERR, "Primary process is missing");
return -1;
}
eth_dev->process_private = (struct pmd_process_private *)
rte_zmalloc(name,
sizeof(struct pmd_process_private),
RTE_CACHE_LINE_SIZE);
if (eth_dev->process_private == NULL) {
MIF_LOG(ERR,
"Failed to alloc memory for process private");
return -1;
}
rte_eth_dev_probing_finish(eth_dev);
return 0;
}
hence as shared in the comment, `one can not create a new memif interface via secondary DPDK process but use the already existing device from primary.
[Answer] as pointed out in the comments DPDK multi Process link does not talk about memif initialization at all.
[Answer] as explained above this is not possible with the current memif PMD code.
**- Do I have to initialize, configure and run memif client eth device on the primary process to be able to run memif client on the secondary process?
[Answer] Yes, using DPDK primary application one has to initialize and configure the memif just like regular DPDK ports.**
[Answer] once the primary dpdk application configures and starts the memif, the secondary application can make use of rx_burst and tx_burst.
Recommendation: As shared in the comment, the ideal way of using memif is between 2 DPDK processes in the following combination