I implementing the Poll Mode Driver(PMD) memif(vdev=net_memif) virtual device in application. Specificly, I have to run memif as a client in secondary process.According to the Dpdk pmd multiprocess application specification, I have to initialize memif vdev at primary process and use the device at secondary process. With this information I can implement the server memif vdev which can run at secondary process.The steps that I have done to make it work like below
./primaryApp --proc-type=primary --file-prefix=pmd1 --vdev=net_memif,role=server,socket=/tmp/memif.sock, socket-abstract=no./secondaryApp --proc-type=auto --file-prefix=pmd1.But in opposite configuration (run memif as client at secondary process), I cannot able to run by implementing the same steps.
I have a question want to ask.
This is the configuration step for memif eth device which runs at secondary process,
if(!rte_eth_dev_get_port_by_name(m_name.c_str(),&m_portId))
std::cout << "Get net_memif eth device successfully port id : " << m_portId << "\n";
else
{
std::cout << "rte_eth_dev_get_port_by_name ERROR \n";
return false;
}
auto eth_count = rte_eth_dev_count_avail();
if(eth_count < 0)
return false;
if(!rte_eth_dev_info_get(m_portId,ðInfo)) {
std::cout << "Get eth dev info successfully\n";
std::cout << " port id : " << m_portId <<"ethInfo : " << ethInfo.driver_name << "\n";
}
else
{
std::cout << "Get eth dev info ERROR\n";
return false;
}
if(!rte_eth_dev_configure(m_portId, m_config.queue_rx_count, m_config.queue_tx_count, ðConfig)) {
std::cout << "Configure eth dev successfully\n";
}
else
{
std::cout << "Configure eth dev ERROR\n";
return false;
}
if(!rte_eth_dev_adjust_nb_rx_tx_desc(m_portId, &m_config.queue_rx_desc, &m_config.queue_tx_desc))
std::cout << "Adjust eth dev rx tx desc successfully\n";
else
{
std::cout << "Adjust eth dev rx tx desc ERROR\n";
return false;
}
for (int i = 0; i < m_config.queue_rx_count; i++) {
if(!rte_eth_rx_queue_setup(m_portId, i, m_config.queue_rx_desc, rte_eth_dev_socket_id(m_portId), ðInfo.default_rxconf, mempool))
std::cout << "Setup rx queue successfully socket id : " << rte_eth_dev_socket_id(m_portId) << std::endl;
else
{
std::cout << "Setup rx queue ERROR\n";
return false;
}
}
for (int i = 0; i < m_config.queue_tx_count; i++) {
if(!rte_eth_tx_queue_setup(m_portId, i, m_config.queue_tx_desc, rte_eth_dev_socket_id(m_portId), ðInfo.default_txconf))
std::cout << "Setup tx queue successfully socket id : " << rte_eth_dev_socket_id(m_portId) << "\n";
else
{
std::cout << "Setup tx queue ERROR\n";
return false;
}
}
then I am starting the memif eth device as fallows,
if( rte_eth_promiscuous_enable(m_portId) != 0) {
std::cout << "rte_eth_promiscuous_enable error\n";
return false;
}
else
std::cout << "rte_eth_promiscuous_enable success \n";
if(!rte_eth_dev_start(m_portId)) {
std::cout << "Eth dev is started \n";
}
else {
std::cout << "Eth dev is start ERROR \n";
return false;
}
Can anybody give me a clue about this issue ? I could not find an information such an implementation and dpdk documentations are not well documented. Any help will be appreciated. The primary process detailed output (it just run rte_eal_init()):
sudo bin/test-memif-dummy --proc-type=primary --file-prefix=pmd2 --vdev=net_memif0,id=0,role=client,socket=/tmp/memif_test.sock,socket-abstract=no --socket-mem=512 --legacy-mem --log-level=eal,8
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Detected lcore 4 as core 0 on socket 0
EAL: Detected lcore 5 as core 1 on socket 0
EAL: Detected lcore 6 as core 2 on socket 0
EAL: Detected lcore 7 as core 3 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Checking presence of .so 'librte_eal.so.21.0'
EAL: Checking presence of .so 'librte_eal.so.21'
EAL: Checking presence of .so 'librte_eal.so'
EAL: Detected static linkage of DPDK
EAL: Ask a virtual area of 0x7000 bytes
EAL: Virtual area found at 0x100000000 (size = 0x7000)
EAL: Multi-process socket /var/run/dpdk/pmd2/mp_socket
EAL: DPAA Bus not present. Skipping.
EAL: VFIO PCI modules not loaded
EAL: Bus dpaa_bus wants IOVA as 'DC'
EAL: Bus fslmc wants IOVA as 'DC'
EAL: Bus pci wants IOVA as 'DC'
EAL: Buses did not request a specific IOVA mode.
EAL: Module /sys/module/rte_kni not found! error 2 (No such file or directory)
EAL: IOMMU is not available, selecting IOVA as PA mode.
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: IOMMU type 1 (Type 1) is supported
EAL: IOMMU type 7 (sPAPR) is not supported
EAL: IOMMU type 8 (No-IOMMU) is not supported
EAL: VFIO support initialized
EAL: Ask a virtual area of 0x2e000 bytes
EAL: Virtual area found at 0x100007000 (size = 0x2e000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 1048576
EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824
EAL: Detected memory type: socket_id:0 hugepage_sz:2097152
EAL: Creating 2 segment lists: n_segs:32 socket_id:0 hugepage_sz:1073741824
EAL: Ask a virtual area of 0x1000 bytes
EAL: Virtual area found at 0x100035000 (size = 0x1000)
EAL: Memseg list allocated at socket 0, page size 0x100000kB
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x140000000 (size = 0x800000000)
EAL: VA reserved for memseg list at 0x140000000, size 800000000
EAL: Ask a virtual area of 0x1000 bytes
EAL: Virtual area found at 0x940000000 (size = 0x1000)
EAL: Memseg list allocated at socket 0, page size 0x100000kB
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x980000000 (size = 0x800000000)
EAL: VA reserved for memseg list at 0x980000000, size 800000000
EAL: Creating 4 segment lists: n_segs:8192 socket_id:0 hugepage_sz:2097152
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x1180000000 (size = 0x61000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x1180200000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x1180200000, size 400000000
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x1580200000 (size = 0x61000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x1580400000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x1580400000, size 400000000
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x1980400000 (size = 0x61000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x1980600000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x1980600000, size 400000000
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x1d80600000 (size = 0x61000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x1d80800000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x1d80800000, size 400000000
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: Hugepage /mnt/hugepages/pmd2map_1 is on socket 0
EAL: Hugepage /mnt/hugepages/pmd2map_0 is on socket 0
EAL: Requesting 1 pages of size 1024MB from socket 0
EAL: Attempting to map 1024M on socket 0
EAL: Allocated 1024M on socket 0
EAL: Added 1024M to heap on socket 0
EAL: TSC frequency is ~2000000 KHz
EAL: Main lcore 0 is ready (tid=7f12244c1c00;cpuset=[0])
EAL: lcore 1 is ready (tid=7f12234ba400;cpuset=[1])
EAL: lcore 2 is ready (tid=7f1222cb9400;cpuset=[2])
EAL: lcore 3 is ready (tid=7f12224b8400;cpuset=[3])
EAL: lcore 4 is ready (tid=7f1221cb7400;cpuset=[4])
EAL: lcore 5 is ready (tid=7f12214b6400;cpuset=[5])
EAL: lcore 6 is ready (tid=7f1220cb5400;cpuset=[6])
EAL: lcore 7 is ready (tid=7f1203fff400;cpuset=[7])
EAL: PCI device 0000:00:1f.6 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:15d8 net_e1000_em
EAL: Not managed by a supported kernel driver, skipped
EAL: No legacy callbacks, legacy socket not created
Application is started
EAL: msg: bus_vdev_mp
EAL: sendmsg: bus_vdev_mp
EAL: reply: bus_vdev_mp
EAL: msg: eal_vfio_mp_sync
EAL: reply: eal_vfio_mp_sync
The secondary process detailed output:
sudo bin/test-memif-client --proc-type=auto --file-prefix=pmd2 --vdev=net_memif0,id=0,role=client,socket=/tmp/memif_test.sock,socket-abstract=no --socket-mem=512 --legacy-mem --log-level=eal,8
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Detected lcore 4 as core 0 on socket 0
EAL: Detected lcore 5 as core 1 on socket 0
EAL: Detected lcore 6 as core 2 on socket 0
EAL: Detected lcore 7 as core 3 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Auto-detected process type: SECONDARY
EAL: Checking presence of .so 'librte_eal.so.21.0'
EAL: Checking presence of .so 'librte_eal.so.21'
EAL: Checking presence of .so 'librte_eal.so'
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/pmd2/mp_socket_77819_23c2a1740b47
EAL: DPAA Bus not present. Skipping.
EAL: VFIO PCI modules not loaded
EAL: request: bus_vdev_mp
EAL: msg: bus_vdev_mp
EAL: msg: bus_vdev_mp
EAL: Bus dpaa_bus wants IOVA as 'DC'
EAL: Bus fslmc wants IOVA as 'DC'
EAL: Bus pci wants IOVA as 'DC'
EAL: Buses did not request a specific IOVA mode.
EAL: Module /sys/module/rte_kni not found! error 2 (No such file or directory)
EAL: IOMMU is not available, selecting IOVA as PA mode.
EAL: Selected IOVA mode 'PA'
EAL: Probing VFIO support...
EAL: request: eal_vfio_mp_sync
EAL: msg: eal_vfio_mp_sync
EAL: VFIO support initialized
EAL: Ask a virtual area of 0x2e000 bytes
EAL: Virtual area found at 0x100007000 (size = 0x2e000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 1048576
EAL: Ask a virtual area of 0x1000 bytes
EAL: Virtual area found at 0x100035000 (size = 0x1000)
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x140000000 (size = 0x800000000)
EAL: VA reserved for memseg list at 0x140000000, size 800000000
EAL: Ask a virtual area of 0x1000 bytes
EAL: Virtual area found at 0x7f8e7b87b000 (size = 0x1000)
EAL: WARNING: Address Space Layout Randomization (ASLR) is enabled in the kernel.
EAL: This may cause issues with mapping memory into secondary processes
EAL: Analysing 2 files
EAL: Main lcore 0 is ready (tid=7f8e77d50c00;cpuset=[0])
EAL: lcore 1 is ready (tid=7f8e76d49400;cpuset=[1])
EAL: lcore 2 is ready (tid=7f8e76548400;cpuset=[2])
EAL: lcore 4 is ready (tid=7f8e75546400;cpuset=[4])
EAL: lcore 3 is ready (tid=7f8e75d47400;cpuset=[3])
EAL: lcore 5 is ready (tid=7f8e74d45400;cpuset=[5])
EAL: lcore 7 is ready (tid=7f8e5f7fe400;cpuset=[7])
EAL: lcore 6 is ready (tid=7f8e5ffff400;cpuset=[6])
EAL: PCI device 0000:00:1f.6 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:15d8 net_e1000_em
EAL: Not managed by a supported kernel driver, skipped
EAL: No legacy callbacks, legacy socket not created
pmd device init is called
Get net_memif eth device successfully port id : 0
Get eth dev info successfully
port id : 0ethInfo : net_memif
Configure eth dev successfully
Adjust eth dev rx tx desc successfully
Setup rx queue successfully socket id : -1
Setup tx queue successfully socket id : -1
rte_eth_promiscuous_enable success
Eth dev is started
Port 0 Link Down
link up error Resource temporarily unavailable
Port 0 Link Down
link up error Resource temporarily unavailable
Port 0 Link Down
link up error Resource temporarily unavailable
Port 0 Link Down
The third application output (running server) :
sudo bin/test-memif-server --proc-type=auto --file-prefix=pmd1 --vdev=net_memif0,id=0,role=server,socket=/tmp/memif_test.sock,socket-abstract=no --socket-mem=512 --log-level=eal,8
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Detected lcore 4 as core 0 on socket 0
EAL: Detected lcore 5 as core 1 on socket 0
EAL: Detected lcore 6 as core 2 on socket 0
EAL: Detected lcore 7 as core 3 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Auto-detected process type: PRIMARY
EAL: Checking presence of .so 'librte_eal.so.21.0'
EAL: Checking presence of .so 'librte_eal.so.21'
EAL: Checking presence of .so 'librte_eal.so'
EAL: Detected static linkage of DPDK
EAL: Ask a virtual area of 0x7000 bytes
EAL: Virtual area found at 0x100000000 (size = 0x7000)
EAL: Multi-process socket /var/run/dpdk/pmd1/mp_socket
EAL: DPAA Bus not present. Skipping.
EAL: VFIO PCI modules not loaded
EAL: Bus dpaa_bus wants IOVA as 'DC'
EAL: Bus fslmc wants IOVA as 'DC'
EAL: Bus pci wants IOVA as 'DC'
EAL: Buses did not request a specific IOVA mode.
EAL: Module /sys/module/rte_kni not found! error 2 (No such file or directory)
EAL: IOMMU is not available, selecting IOVA as PA mode.
EAL: Selected IOVA mode 'PA'
EAL: No free hugepages reported in hugepages-2048kB
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: IOMMU type 1 (Type 1) is supported
EAL: IOMMU type 7 (sPAPR) is not supported
EAL: IOMMU type 8 (No-IOMMU) is not supported
EAL: VFIO support initialized
EAL: Ask a virtual area of 0x2e000 bytes
EAL: Virtual area found at 0x100007000 (size = 0x2e000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 1048576
EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824
EAL: Detected memory type: socket_id:0 hugepage_sz:2097152
EAL: Creating 2 segment lists: n_segs:32 socket_id:0 hugepage_sz:1073741824
EAL: Ask a virtual area of 0x1000 bytes
EAL: Virtual area found at 0x100035000 (size = 0x1000)
EAL: Memseg list allocated at socket 0, page size 0x100000kB
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x140000000 (size = 0x800000000)
EAL: VA reserved for memseg list at 0x140000000, size 800000000
EAL: Ask a virtual area of 0x1000 bytes
EAL: Virtual area found at 0x940000000 (size = 0x1000)
EAL: Memseg list allocated at socket 0, page size 0x100000kB
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x980000000 (size = 0x800000000)
EAL: VA reserved for memseg list at 0x980000000, size 800000000
EAL: Creating 4 segment lists: n_segs:8192 socket_id:0 hugepage_sz:2097152
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x1180000000 (size = 0x61000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x1180200000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x1180200000, size 400000000
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x1580200000 (size = 0x61000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x1580400000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x1580400000, size 400000000
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x1980400000 (size = 0x61000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x1980600000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x1980600000, size 400000000
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x1d80600000 (size = 0x61000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x1d80800000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x1d80800000, size 400000000
EAL: Allocating 1 pages of size 1024M on socket 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: Added 1024M to heap on socket 0
EAL: TSC frequency is ~2000000 KHz
EAL: Main lcore 0 is ready (tid=7fb885decc00;cpuset=[0])
EAL: lcore 1 is ready (tid=7fb884de5400;cpuset=[1])
EAL: lcore 3 is ready (tid=7fb87f7fe400;cpuset=[3])
EAL: lcore 2 is ready (tid=7fb87ffff400;cpuset=[2])
EAL: lcore 5 is ready (tid=7fb87e7fc400;cpuset=[5])
EAL: lcore 4 is ready (tid=7fb87effd400;cpuset=[4])
EAL: lcore 7 is ready (tid=7fb87d7fa400;cpuset=[7])
EAL: lcore 6 is ready (tid=7fb87dffb400;cpuset=[6])
EAL: PCI device 0000:00:1f.6 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:15d8 net_e1000_em
EAL: Not managed by a supported kernel driver, skipped
EAL: No legacy callbacks, legacy socket not created
8191
pmd device init is called
Get net_memif eth device successfully port id : 0
Get eth dev info successfully
port id : 0ethInfo : net_memif
Configure eth dev successfully
Adjust eth dev rx tx desc successfully
Setup rx queue successfully socket id : -1
Setup tx queue successfully socket id : -1
rte_eth_promiscuous_enable success
Eth dev is started
Port 0 Link Down
Port 0 Link Down
Port 0 Link Down
Port 0 Link Up - speed 10000 Mbps - full-duplex
Application is started
Server receiving
There are some gaps in understanding DPDK Primary-Secondary process and how memif PMD works in DPDK. Let me try to explain
[Answer] based on the code snippet, memif PMD is getting used in custom applications. There is no code snippet indicating memif PMD is modified or enhanced for the requirement.
[Answer] yes, memif can be used in the secondary process. That is, to a primary process that creates a memif interface as server or client; a secondary process can be attached. From the secondary process one can allow rx_burst and tx_burst too. But the memif device instance is created by the primary to be accessed by secondary.
Can one create a new memif interface in secondary? Answer is NO, because of the memif PMD probe code always checks process type as secondary and tries to attach to memif interface in secondary.
File: drivers/net/memif/rte_eth_memif.c
Function: rte_pmd_memif_probe
Code:
if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
eth_dev = rte_eth_dev_attach_secondary(name);
if (!eth_dev) {
MIF_LOG(ERR, "Failed to probe %s", name);
return -1;
}
eth_dev->dev_ops = &ops;
eth_dev->device = &vdev->device;
eth_dev->rx_pkt_burst = eth_memif_rx;
eth_dev->tx_pkt_burst = eth_memif_tx;
if (!rte_eal_primary_proc_alive(NULL)) {
MIF_LOG(ERR, "Primary process is missing");
return -1;
}
eth_dev->process_private = (struct pmd_process_private *)
rte_zmalloc(name,
sizeof(struct pmd_process_private),
RTE_CACHE_LINE_SIZE);
if (eth_dev->process_private == NULL) {
MIF_LOG(ERR,
"Failed to alloc memory for process private");
return -1;
}
rte_eth_dev_probing_finish(eth_dev);
return 0;
}
hence as shared in the comment, `one can not create a new memif interface via secondary DPDK process but use the already existing device from primary.
[Answer] as pointed out in the comments DPDK multi Process link does not talk about memif initialization at all.
[Answer] as explained above this is not possible with the current memif PMD code.
**- Do I have to initialize, configure and run memif client eth device on the primary process to be able to run memif client on the secondary process?
[Answer] Yes, using DPDK primary application one has to initialize and configure the memif just like regular DPDK ports.**
[Answer] once the primary dpdk application configures and starts the memif, the secondary application can make use of rx_burst and tx_burst.
Recommendation: As shared in the comment, the ideal way of using memif is between 2 DPDK processes in the following combination