We have a use-case that has one RX thread and two TX threads operating on same port. The RX thread uses the rte_pktmbuf_free to free the mbuf after processing received packets. The TX threads allocate mbuf using rte_pktmbuf_alloc call and transmit on the port.
When using same pool for RX descriptor init, and for mbuf allocation in TX threads, we see that sometimes there are unexpected mbuf leaks and allocation failures. If we use separate pools for RX and each of the TX threads, then we do not see these issues
We have not used any flags in the mempool_create call. The DPDK documentation seems to indicate mempool operations are thread safe?
We are using DPDK version: 19.11
Are we supposed to create unique mempool for each of the RX and the two TX threads?
EDIT1: Code snippet added as requested by Vipin
mbuf_pool = rte_pktmbuf_pool_create("fast_pkt", 8192,
256, 0, RTE_MBUF_DEFAULT_BUF_SIZE, rte_socket_id());
/* Configure the Ethernet device. */
retval = rte_eth_dev_configure(port, rx_rings, tx_rings, &port_conf);
/* Allocate and set up 1 RX queue per Ethernet port. */
retval = rte_eth_rx_queue_setup(port, q, 128,
rte_eth_dev_socket_id(port), NULL, mbuf_pool);
/* Allocate and set up 2 TX queue */
for (q = 0; q < tx_rings; q++) {
retval = rte_eth_tx_queue_setup(port, q, 512,
rte_eth_dev_socket_id(port), NULL);
}
/* Start the Ethernet port. */
retval = rte_eth_dev_start(port);
mbuf_pool = rte_mempool_lookup("fast_pkt");
/* Create rx thread attach it to vCPU #4 */
rte_eal_remote_launch( main_rx, NULL, 4 );
/* create two tx threads and assign affinity to different vCPUs*/
pthread_create(&thread_id, NULL, tx_1, NULL );
CPU_ZERO(&cpuset);
CPU_SET(5, &cpuset);
pthread_setaffinity_np(thread_id, sizeof(cpuset), &cpuset);
pthread_create(&thread_id, NULL, tx_2, NULL );
CPU_ZERO(&cpuset);
CPU_SET(6, &cpuset);
pthread_setaffinity_np(thread_id, sizeof(cpuset), &cpuset);
for (;; )
{
nb_rx = rte_eth_rx_burst(port , 0, pkts_burst, 32);
for ( i = 0; i < nb_rx ; i++ )
{
process_packet( pkts_burst[ i] );
rte_pktmbuf_free(pkts_burst[ i]);
}
}
tx_q = <thread_sepcific queue ID>
for ( ;; )
{
/* Receive Internal message for TX*/
get_packet ( work_buffer [Burst]);
for ( iterate through each index in work_buffer )
{
tx_mbuf[i] = rte_pktmbuf_alloc(mbuf_pool);
/* combination of following calls to construct packet */
rte_pktmbuf_mtod( .. );
/* follow above call with copy from work buffer */
rte_pktmbuf_prepend( .. );
rte_pktmbuf_append( .. );
}
rte_eth_tx_burst(start_port_id, tx_q, tx_mbuf, num_burst)
/* loop through packets not processed by rte_eth_tx_burst */
for(.. )
{
rte_pktmbuf_free (... );
}
}
[EDIT-1 answer shared below is a converged questions shared and clarification provided]
is mempool operations are thread-safe? - (answer) Yes it is with respect to Lcore DPDK threads.Are we supposed to create unique mempool for each of the RX and the two TX threads? - (answer) No, one can safely use the same mempool across different ports and port-queue pairs alike.there is mbuf leak and depletion of mbuf_pool (sample code flow shared in question) - (answer) after correcting the code, it has been running over12 hours without leaks.we wanted to understand if the rte_pktmbuf_alloc and rte_pktmbuf_free are intended to be thread-safe or not - (answer) YesOur RX thread uses "rte_eal_remote_launch" whereas our TX threads are created directly using pthread_create from the application. Do you think that makes a difference to thread safety? Should our TX threads also be launched using "rte_eal_remote_launch"? - (answer) yeswe should not have multiple threads (I.e., pthreads) allocated to same lcore and accessing same memory pool. - (answer) Yes, that is the right understandingreason: DPDK lcore threads internally has more things done than thread create and set affinity. Please refer DPDK EAL Documentation for clarity. based on options selected for rte_eal_init
N DPDK lcore threads.index, which is rte_lcore_idrte_pktmbuf_alloc it uses the id to fetch the cache mempool objects. using the same id on multiple threads violates the lockless model and can cause a hazard if multiple requests come concurrently.hence rewriting the code snippet with DPDK example skeleton in primary-secondary mode with the right API calls does not produce the memory leaks. Stats showcasing the run shared below
######################## NIC statistics for port 0 ########################
RX-packets: 15048531 RX-errors: 0 RX-bytes: 1123238216
RX-nombuf: 0
TX-packets: 15048504 TX-errors: 0 TX-bytes: 902910240
############################################################################
######################## NIC statistics for port 0 ########################
RX-packets: 23424430063 RX-errors: 0 RX-bytes: 1405686130136
RX-nombuf: 0
TX-packets: 23424430036 TX-errors: 0 TX-bytes: 1405465802032
############################################################################
RX-nombuf easily shows if the PMD was not able to get packets due to absence of mbuf
mempool info
========== show - MEMPOOL ==========
mempool <MBUF_POOL>@0x11ff9fad00
flags=10
socket_id=1
pool=0x11ff9eab00
iova=0xeff9fad00
nb_mem_chunks=1
size=8191
populated_size=8191
header_size=64
elt_size=2304
trailer_size=0
total_obj_size=2368
private_data_size=64
ops_index=7
ops_name: <ring_mp_mc>
avg bytes/object=2368.289098
internal cache infos:
cache_size=250
cache_count[32]=171
cache_count[34]=90
total_cache_count=261
common_pool_count=2874
note: above stats fetched via dpdk-proc-info utility.