The facilities we can use in Unices systems for asynchronous I/O alerts, such as epoll on Linux, kqueue on BSD systems and Solaris /dev/poll or I/O Ports, all let the user to specify a pointer to be associated to the file descriptor the user want to receive I/O alerts.
Usually in this pointer the user specify the pointer to a structure which will abstract a file descriptor (such as a "Stream" structure, or things like that), and the user will allocate a new structure every time a new file descriptor is open.
E.g. struct stream { int fd; int flags; callback_t on_read_fn; /* ... */ };
Now, my question is: how to safely deallocate this structure the user allocate in a multithreaded envinronment?
I ask this, because of the nature of epoll/kqueue/etc: You usually have a thread which "downloads" from the kernel a vector of events, containing file descriptors which have some I/O readiness, and an user pointer associated with that file descriptor.
Now, let's consider I have 2 threads: T1 which downloads those events and process them, e.g. calling stream->on_read_fn(); etc, and T2 which simply runs user code, user events, and stuff like that.
If T2 wants to close a file descriptor, simply does close(stream->fd); and T1 will not receive any I/O alerts for that fd anymore, so its safe to deallocate the stream structure there.
But WHAT IF the T1 thread has already downloaded that very same file descriptor in the vector of events it is processing right now, and it hasn't processed that file descriptor yet?
If T1 is scheduled BEFORE T2, it will be OK, but if T2 is scheduled BEFORE T1, it will close the file descriptor and deallocate the stream structure, so the thread T1, when it will process that file descriptor, will have a user associated pointer, pointing to an already deallocated structure! Of course this will crash badly.
My point is that T2 will never know IF the thread T1 downloaded some I/O alerts for that specific file descriptor, neither T2 can forecast IF T1 will ever download some I/O alerts or not at all!
This is very tricky, and its making my head spin. Any thoughts? When is safe to deallocate the user specified pointer in this scenario?
NOTE: a friend of mine suggested to remove the file descriptor from the epoll/kqueue queue BEFORE calling close(2) on it. This is right, and this is what I do right now, but this won't solve the problem, because T2 can remove the file descriptor from the epoll/kqueue queue, but this won't assure an I/O event for that file descriptor hasn't been already "downloaded" from the kernel and will be processed soon by thread T1.
i had the exact similar problem, that's why in the new linux kernel proposals, someone (cant' remember the name) suggested to implement a DISABLED status for the FD so you can skip processing if it has been deallocated by another thread.
Personally, I moved from a multithreaded epool calls, to a single thread that epool() on the FDs , and then schedule the events to multiple threads. Object themself inside are reference counted, and collected later by a garbage collector. Works quite good honestly and without noticeable degradation against the multithreaded epool solution...
* EDITED *
Also, I've investigated another way to close the FD from the same thread than handles epool by creating a std::set protected by a mutex, and filled in by consumer threads as long as the FD needs to be closed. This worked quite good too.