How to make an unblocking connection from a server in Twisted?

Question

I am implementing an API with the following basic structure:

1. Run serverA
2. Client connects to serverA and sends data
3. ServerA processes the data and sends it on to serverB
4. ServerB replies to serverA
5. ServerA responds to client's request based on the input received from serverB

So far I have tried two solutions:

1) Create a standard non-twisted TCP connection using httplib to process the request from serverA to serverB. This however effectively blocks the server for the duration of the httplib call.

2) Create a second class inheriting from protocol.Protocol and use

factory = protocol.ClientFactory()
factory.protocol = Authenticate
reactor.connectSSL("localhost",31337,factory, ssl.ClientContextFactory())

to create the connection between serverA and serverB. However when doing this, I don't seem to be able to access the original client-to-serverA connection from within the callbacks of the request class.

What would be the correct way to handle such a setting in Twisted?

Answer 1

The "client-to-serverA" connection is represented by a protocol instance associated with a transport. These are both regular old Python objects, so you can do things like pass them as arguments to functions or class initializers, or set them as attributes on other objects.

For example, if you have ClientToServerAProtocol with a fetchServerBData method which is invoked in response to some bytes being received from the client, you might write it something like this:

class ClientToServerAProtocol(Protocol):
    ...
    def fetchServerBData(self, anArg):
        factory = protocol.ClientFactory()
        factory.protocol = Authenticate
        factory.clientToServerAProtocol = self
        reactor.connectSSL("localhost",31337, factory, ssl.ClientContextFactory())

Since ClientFactory sets itself as the factory attribute on any protocol it creates, the Authenticate instance which will result from this will be able to say `self.factory.clientToServerAProtocol and get a reference to that "client-to-serverA" connection.

There are lots of variations on this approach. Here's another one, using the more recently introduced endpoint API:

from twisted.internet.endpoints import SSL4ClientEndpoint

class ClientToServerAProtocol(Protocol):
    ...
    def fetchServerBData(self, anArg):
        e = SSL4ClientEndpoint(reactor, "localhost", 31337, ssl.ClientContextFactory()) 
        factory = protocol.Factory()
        factory.protocol = Authenticate
        connectDeferred = e.connect(factory)
        def connected(authProto):
            authProto.doSomethingWith(self)
        connectDeferred.addCallback(connected)

Same basic idea here - use self to give a reference to the "client-to-serverA" connection you're interested in to the Authenticate protocol. Here, I used a nested function to ''close over'' self. That's just another of the many options you have for getting references into the right part of your program.