Search code examples
c++linuxperformancemessaging

linux high performance message passing between threads in c++

say i have a process which spawns 2 threads

the first thread listens in a tight loop for packet events on a UDP port. the second thread is to receive the bytes contained in this UDP packet and parse it/do stuff.

can anyone recommend a faster method than using linux message queues? i think they are slow because they are copying the bytes upon writing to the queue and copying them again upon reading from the queue

i am aware of the zeromq library but is there a slick way to do this without that overhead? i realize i could just use a tcp/ip socket between the two threads for simple queued communication but is there a faster way?

i am thinking maybe a ring buffer in memory that is shared between the threads and a mutex used to control a pointer to the most recently updated element?

anyones thoughts here?

Solution

  • The most efficient method that I can think of would use one linked list, one mutex, and one condition variable:

    • Thread A:

      1. Allocate a udp-packet-buffer object (the object should contain previous and next pointers, and an array of bytes to store UDP data in)
      2. recv() UDP data into the object's byte-array
      3. lock the mutex
      4. append the udp-packet-buffer object to tail of the linked-list
      5. unlock the mutex
      6. signal the condition variable
      7. goto 1

    • Thread B:

      1. wait on the condition variable (until the condition variable is signaled)
      2. lock the mutex
      3. pop the next udp-packet-buffer off the head of the linked-list
      4. unlock the mutex
      5. parse/handle the UDP data in the udp-packet-buffer object
      6. delete the udp-packet-buffer object
      7. goto 1

    That gives you a zero-copy communications mechanism. The time spent holding the mutex is minimal, so contention should be low also.

    If you wanted to optimize a bit further, you could have thread B grab all of the items out of the linked-list at once instead of just popping off one at a time -- something that is possible to do in O(1) time with a linked list. And then have thread A signal the condition variable only if the linked list was empty just before it appended its latest udp-packet-buffer object. That would cut down on the number of times the threads have to lock/unlock/signal when under heavy load.