Twisted many inlineCallbacks at once

Question

Short brief about my situation: I'm writing a server (twisted powered) which handles WebSocket connections with multiple clients (over 1000). Messages send from server to client are handled via Redis pub/subinterface (because messages can be applied via REST) in that flow:

• REST appends command to the client and publishes,
• twisted is getting poked because it subscribes that redis channel,
• the message is added to the client queue and waits for further processing

Now, as client connects, gets registered I'm launching inlineCallback for each client to sweep throught the queue, like this:

    @inlineCallbacks
    def client_queue_handler(self, uuid):
        queue = self.send_queue[uuid]
        client = self.get_client(uuid)
        while True:
            uniqueID = yield queue.get()          
            client_context = self.redis_handler.get_single(uuid)
            msg_context = next(iter([msg
                                for msg in client_context
                                if msg['unique'] == unique]),
                                    None)

            client.sendMessage(msg_context)

As I said previously, many clients may connect. Is this perfectly fine, that each client has it's own inlineCallback which performs an infinite loop? As far as I know, twisted has customizable thread pool limit. What will happen if there will be more clients (inlineCallbacks) than threads in threadpool? Will queue.get() block/sleep that "virtual thread" and pass control to the other one? Maybe one "global" thread which sweeps all clients is a better option?

Answer 1

inlineCallbacks doesn't start any OS threads. It's just a different interface to using Deferred. A Deferred is just an API for dealing with callbacks.

queue.get() returns a Deferred. When you yield it, inlineCallbacks internally adds a callback to it and your function remains suspended. When the callback fires, inlineCallbacks resumes your function with the value passed to the callback - which is the "result" of the Deferred you yielded.

All that's happening is some Deferred objects are being created and some callbacks are being added to them. Somewhere inside the implementation of your redis client, some event sources are "firing" the Deferred with a result which starts the process of calling its callbacks.

You can have as many of these: * as you have system memory to hold * as the redis client can keep track of at a time

I don't know the details of how your redis client is implemented. If it has to open a socket per queue then you'll probably be limited to the number of file descriptors you can open or the number of sockets your system can support. These numbers will be somewhere in the tens of thousands and when you bump into them, there are tricks you can deploy to raise the limit further.

If it doesn't have to open a socket per queue (for example, if it can multiplex notification for all the queues over a single socket) then it probably has a limit that's much, much higher (maybe imposed by algorithmic complexity of the slowest part of its implementation).