Search code examples
pythontcpbuffertwisted

Flushing Twisted's write buffer

Twisted's Connection class inherits the write method from the abstract class FileDescriptor. As you can see here, the class has a buffer which is not flushed until the total number of buffered bytes is greater than bufferSize (attribute in class FileDescriptor).

For my specific needs, I'd like to write data to the socket as soon as possible, so I would like to avoid buffering them at any point of the transmission. I have set TCP_NODELAY to the socket but data is still buffered at Twisted's write call.

So, my approach to tackle this is to call FileDescriptor.doWrite method, which will try to write any data in the buffer (see source). I call doWrite always after write as follows:

...
self.transport.write(data)
self.transport.doWrite()
...

The workaround seems to be working fine but, from time to time, the following bug comes to the surface:

    self.transport.doWrite()  # Flush twisted buffer
File "/usr/local/lib/python2.7/dist-packages/Twisted-14.0.2-py2.7-linux-x86_64.egg/twisted/internet/abstract.py", line 270, in doWrite
    self.stopWriting()
File "/usr/local/lib/python2.7/dist-packages/Twisted-14.0.2-py2.7-linux-x86_64.egg/twisted/internet/abstract.py", line 429, in stopWriting
    self.reactor.removeWriter(self)
File "/usr/local/lib/python2.7/dist-packages/Twisted-14.0.2-py2.7-linux-x86_64.egg/twisted/internet/epollreactor.py", line 344, in removeWriter
    EPOLLOUT, EPOLLIN)
File "/usr/local/lib/python2.7/dist-packages/Twisted-14.0.2-py2.7-linux-x86_64.egg/twisted/internet/epollreactor.py", line 322, in _remove
   primary.remove(fd)
KeyError: 11

The problem is that doWrite calls stopWriting, which attempts to remove itself (the Connection object) from the writers list in the reactor. The exception raises because it cannot find this object in the list.

It's strange because write should register the reader in case the data argument is not None (I thought the problem was that data sometimes was None, so I included write & doWrite under the condition of data being present). So, I decided to just catch the KeyError exception and ignore it as follows:

...
if data:
    self.transport.write(data)
    try:
        self.transport.doWrite()
    except KeyError:
        pass
...

Yet, the exception keeps popping and even closes the TCP connection.

I don't know where and why the exception is raised now that I catch it in doWrite. Also, I don't know what are the side-effects of this hack. It might be that I'm oversimplifying the whole Twisted's file descriptor paradigm (I don't understand it enough yet). I also don't know if this is the actual expected behaviour, shouldn't Twisted check whether the element is in the list of writers before removing it from the list in any case?

Solution

  • doWrite isn't there for you to call. It is part of the interface between the transport and the reactor. It is not part of the interface between the transport and the protocol.

    FileDescriptor does not buffer writes until a certain number of bytes have been written. It only buffers writes (at most) until the reactor gets a chance to run again. Twisted is a cooperative multi-tasking system. Only one thing happens at a time. If the one thing that's happening is your application code running then the reactor isn't taking care of its job (like writing bytes to file descriptors). Each thing has to wait for the other to finish before it can start.

    Whatever led you to the conclusion that there's a size-based write buffer was a mistake. Whatever problem you're trying to solve, the solution doesn't look like this.