Can a socket become writeable after an EWOULDBLOCK but before an epoll_wait?

Question

I have a reactor based design where a number of socket file descriptors are polled in an epoll_wait.

In order to optimise for latency not throughput, when writing to a socket we attempt to write directly to the file descriptor immediately.

In the event of EWOULDBLOCK we then buffer the unwritten data and then wait for the socket to become writeable again.

The next time our epoll_wait returns our file descriptor as writeable, we then send from the buffered data.

The question is, can the socket become writeable after we receive an EWOULDBLOCK but before we call epoll_wait?

That is, if we do 2 write calls, where the first write is partially written and the remainder buffered, and then do the second write, attempting to write immediately, could the socket become writeable between the first and second write call, and we end up with out-of-order data being received on the far side?

Answer 1

The question is, can the socket become writeable after we receive an EWOULDBLOCK but before we call epoll_wait?

Yes, it can. At the time of the EWOULDBLOCK, the socket buffer is full. But as soon as the peer ACKs some data and space clears up in the socket buffer, the socket becomes writable again. That happens in the background outside of your code, so the socket can become writable at any time.

That is, if we do 2 write calls, where the first write is partially written and the remainder buffered, and then do the second write, attempting to write immediately, could the socket become writeable between the first and second write call, and we end up with out-of-order data being received on the far side?

Yes, it can. And this would be a bug in your code logic, because you attempted to write new data to the socket before you finished writing any buffered data that was still waiting to be sent.

When you detect EWOULDBLOCK and start buffering data, stop calling write() for any new data until your buffer has been completely cleared. If you need to send new data, and there is already data in your buffer, simply append the new data to the end of your buffer and move on, don't even call write() at all. Whenever the socket is reported as writable, check if your buffer has any pending data in it, and if so then send as much as you can from the front of the buffer, removing successfully sent data as you go, until the buffer is cleared or EWOULDBLOCK is reported again.