Search code examples
multithreadingepollepollet

Multithreaded TCP listener with epoll and EPOLLET in C

I want to write a multithreaded TCP listener using epoll and EPOLLET.

I have seen that there are several possibilities:

  1. Each thread has its own epoll fd, does bind() using SO_REUSEPORT (but only up Linux kernel 3.9) and handles its own connections. EPOLLONESHOT would not be needed in this case because each thread handles its own file descriptors.

  2. There is a main thread which accepts connections and several worker threads which handle those connections. Each worker thread has its own epoll fd. How can the main thread in this case fairly distribute the connections among the worker threads? It could add the file descriptor to the epoll fd of another thread using a round-robin fashion (but it might happen that that particular thread is still busy handling another connection). Or connections could be added to a global queue, and the main thread would use pthread_cond_signal(), but then we need a mutex and a condition variable.

  3. There is a main thread which accepts connections and several worker threads which handle those connections. There is a global epoll fd, EPOLLONESHOT would then be needed in this case, so not all the threads get woken up for the same event.

I know that, if EPOLLET is used, once I get notified about an event, I have to drain the fd, until I get EAGAIN.

If the socket option SO_REUSEPORT is not supported (older kernel), which option would be the best?

Solution
    1. A solution without using SO_REUSEPORT would be to have a common epoll fd and a common listener which are shared among all the threads. EPOLLONESHOT is required so only one thread handles the events for a certain fd at a time.