Search code examples
linuxmultithreadingparallel-processingpthreadsreal-time

How to run short asynchronous tasks efficiently under Linux?

I'm trying to meet a soft real-time requirement where a task needs to complete in < 1 ms under Linux. Currently I'm using pthreads with 4 - 8 threads to try and achieve this, but it seems that the overheads and latencies with pthreads under Linux are not well suited to short duration tasks (~100 µs latency each for both pthread_create and pthread_join + some weird non-deterministic behaviour while the threads are running which can add 100 - 200 µs more).

So I'm wondering whether there is any other way to run short async tasks reliably, with reasonably low latency. The tasks are typically < 500 µs and I need them all to complete within 1 ms. Might I be able to use kernel threads (kthreads) directly somehow (e.g. using shared memory for task data) ? Or perhaps something based on interrupts ?

Background information

I have tried playing around with the scheduling options with pthreads on Linux. SCHED_FIFO and SCHED_RR tend to make things worse, regardless of thread priority. Setting thread affinity (pthread_set_affinity_np) helps a little though, as it reduces thread migration between cores.

The current code has also been tested on Mac OS X (which is based on BSD and the Mach kernel) - it works fine with pthreads, and it easily meets the < 1 ms requirement.

It seems that pthreads on Linux is not so well optimised for short duration threads. According to this paper: "The Linux Scheduler: a Decade of Wasted Cores" - there are a number of problems with pthreads on Linux, which seem to have arisen due to the haphazard way in which multi-core support was introduced into the Linux scheduler. My problem does not seem to be related to any of the four problems identified in the paper, but it does suggest that threads on Linux may be something of a curate's egg.

Solution

  • task needs to complete in < 1 ms under Linux

    Look, it seems to be not a very strict requirement. There are lots of low-latency financial software that face with even tougher requirements.

    There are lots of advice on the Internet on to write low-latency software, for example:

    1. https://access.redhat.com/sites/default/files/attachments/2012_perf_brief-low_latency_tuning_for_rhel6_0.pdf,
    2. https://codedependents.com/2014/01/27/11-best-practices-for-low-latency-systems,
    3. https://www.quora.com/How-does-one-become-a-low-latency-programmer,
    4. https://www.linkedin.com/pulse/mentor-low-latency-c-code-joe-ellsworth.

    Since you mentioned creating and deleting threads in your question I think you break one of these recomendation (Keep context switches to a minimum). I think you should keep one or a few threads busy waiting, not creating and joining.