Search code examples
linuxmultithreadingpthreadswaitformultipleobjects

Best practice for waiting for events on multiple threads in Linux (like WaitForMultipleObjects)

In Windows there is the API WaitForMultipleObjects which will, if one event is registered in many threads, only wake one thread if the event occurs. I now have to port an application that uses this in its threadpool and I am looking for the best practive to do this in Linux.

I am aware of epoll which can wait for fds (which i can create with pipe), but waiting on one FD in multiple threads may wake every thread on event when only one is needed.

What would be the best practice to implement this behaviour on Linux? I really dont want to split up an event to have as many FDs as there are worker threads, as this may hit the FD limit on some systems as I have many events (which all would be split up).

What I thought about is create 1 master thread that will delegate work to an available worker (or queue the task if all workers are working), but that would mean that I have one additional context switch (and thus giving up computation time) as the master will wake up and then wake up another worker. I would do this if there is no other possibility to cleanly implement this. Unfortunately I cannot get rid of the current architecture so I need to get around this.

Is there any API that would be applicable for this kind of problem?

Solution

  • epoll() is the correct solution, although you could consider using eventfd() file descriptors rather than pipe() file descriptors for the event signalling. See this text from the epoll(7) man page:

    If multiple threads (or processes, if child processes have inherited the epoll file descriptor across fork(2)) are blocked in epoll_wait(2) waiting on the same the same epoll file descriptor and a file descriptor in the interest list that is marked for edge-triggered (EPOLLET) notification becomes ready, just one of the threads (or processes) is awoken from epoll_wait(2). This provides a useful optimization for avoiding "thundering herd" wake-ups in some scenarios.

    So to get this single-wakeup behaviour, you have to be calling epoll_wait() in each thread on the same epoll descriptor, and you have to have registered your event-notifying file descriptors in the epoll set as edge-triggered.