Cannot allocate memory :Failed to create packet memory pool (rte_pktmbuf_pool_create failed) - for port_id 0

I have upgraded DPDK from 17.02 to 21.11. RPM build was built and installed successfully. While running the custom application I saw the following error:

Cannot allocate memory#012ms_dpdk::port::port: Failed to create packet memory pool (rte_pktmbuf_pool_create failed) - for port_id

Function call parameters : rte_pktmbuf_pool_create(port-0,267008,32,0,2176,0)

I have added std::string msg = rte_strerror(rte_errno); in error logs and it gives the output as Cannot allocate memory

LDD output shows the libraries are linked properly and there are no "no found" entries.

ldd /opt/NETAwss/proxies/proxy | grep "buf"
        librte_mbuf.so.22 => /lib64/librte_mbuf.so.22 (0x00007f795873f000)

ldd /opt/NETAwss/proxies/proxy | grep "pool"
        librte_mempool_ring.so.22 => /lib64/librte_mempool_ring.so.22 (0x00007f7a1da3f000)
        librte_mempool.so.22 => /lib64/librte_mempool.so.22 (0x00007f7a1da09000)

igb_uio is also loaded successfully.

lsmod | grep uio

igb_uio                 4190  1
uio                     8202  3 igb_uio

cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages 512

grep Huge /proc/meminfo

AnonHugePages:    983040 kB

ShmemHugePages:        0 kB

HugePages_Total:     512

HugePages_Free:      511

HugePages_Rsvd:        0

HugePages_Surp:        0

Hugepagesize:       2048 kB

When I run dpdk-testpmd it seems to be working fine. Below is the output of the test application.

  ./dpdk-testpmd
 EAL: Detected CPU lcores: 2
 EAL: Detected NUMA nodes: 1
 EAL: Detected static linkage of DPDK
 EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
 EAL: Selected IOVA mode 'PA'
 EAL: Probe PCI driver: net_vmxnet3 (15ad:7b0) device: 0000:13:00.0 (socket 0)
 TELEMETRY: No legacy callbacks, legacy socket not created
 testpmd: create a new mbuf pool <mb_pool_0>: n=155456, size=2176, socket=0
 testpmd: preferred mempool ops selected: ring_mp_mc

 Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.

 Configuring Port 0 (socket 0)
 Port 0: 00:50:56:88:9A:43
 Checking link statuses...
 Done
 No commandline core given, start packet forwarding
 io packet forwarding - ports=1 - cores=1 - streams=1 - NUMA support enabled, MP allocation mode: native
 Logical Core 1 (socket 0) forwards packets on 1 streams:
   RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00

   io packet forwarding packets/burst=32
   nb forwarding cores=1 - nb forwarding ports=1
   port 0: RX queue number: 1 Tx queue number: 1
     Rx offloads=0x0 Tx offloads=0x0
     RX queue: 0
       RX desc=0 - RX free threshold=0
       RX threshold registers: pthresh=0 hthresh=0  wthresh=0
       RX Offloads=0x0
     TX queue: 0
       TX desc=0 - TX free threshold=0
       TX threshold registers: pthresh=0 hthresh=0  wthresh=0
       TX offloads=0x0 - TX RS bit threshold=0
 Press enter to exit

 Telling cores to stop...
 Waiting for lcores to finish...

   ---------------------- Forward statistics for port 0  ----------------------
   RX-packets: 2              RX-dropped: 0             RX-total: 2
   TX-packets: 2              TX-dropped: 0             TX-total: 2
   ----------------------------------------------------------------------------

   +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
   RX-packets: 2              RX-dropped: 0             RX-total: 2
   TX-packets: 2              TX-dropped: 0             TX-total: 2
   ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

 Done.

 Stopping port 0...
 Stopping ports...
 Done

 Shutting down port 0...
 Closing ports...
 Port 0 is closed
 Done

 Bye...

I am not able to figure out the root cause of this error. Any help is appreciated. Thanks

Solution

Memory allocation failure happens by moving from DPDK 17.02 to 21.11. This is expected for fixed 512 * 2MB and memory requirements from custom application.
DPDK 21.11 introduces new features like telemetry, fb_arrary, MP communication sockets, service cores, which requires more internal memory allocation (not everything is from HEAP region but hugepage).
rte_pktmbuf_pool_create tries to create (267008 * 2176 + additional place holder) is about 0.8GB.

hence with the above new memory model and services, a total huge page potential shoots over 1GB MMAPED area. Currently, the Huge pages allocated in the system are 512 * 2MB only.

Solutions:

reduce the number of MBUF from 267008 to a lower value like 200000 to satisfy the memory requirement.
Increase the number of available huge pages from 512 to 600
use the new EAL to use legacy memory, no telemetry, no multiprocess, no service cores, to reduce memory footprint.
use real arg --socket-mem or -m, to fix the memory allocations.

Note: the RPM package was not initially housing libdpdk.pc. This is required for obtaining platform-specific CFLAGS and LDFLAGS.