Django: communicate with TCP server (with twisted?)

Question

I have a django application, that needs to talk to a remote TCP server. This server will send packages and depending on what the package is, I need add entries to the database and inform other parts of the application. I also need to actively send requests to the TCP server, for instance when the user navigates to a certain page, I want to subscribe to a certain stream on the TCP server. So communication in both directions need to work.

So far, I use the following solution:

I wrote a custom Django command, that I can start with

python manage.py listen

This command will start a twisted socket server with reactor.connectTCP(IP, PORT, factory) and since it is a django command, I will have access to the database and all the other parts of my application.

But since I also want to be able to send something to the TCP server, triggered by a certain django view, I have an additional socket server, that starts within my twisted application by reactor.listenTCP(PORT, server_factory).

To this server, I will then connect directly in my django application, within a new thread:

class MSocket:

    def __init__(self):
        self.stopped = False
        self.socket = None
        self.queue = []
        self.process = start_new_thread(self.__connect__, ())
        atexit.register(self.terminate)


    def terminate(self):
        self.stopped = True
        try:
            self.socket.close()
        except:
            pass

    def __connect__(self):
        if self.stopped:
            return
        attempts = 0
        self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        while True and not self.stopped:
            try:
                print "Connecting to Socket Server..."
                self.socket.connect(("127.0.0.1", settings.SOCKET_PORT))
                print "Connection Successful!"
                for msg in self.queue:
                    self.socket.send(msg)
                self.queue = []
                break
            except:
                pause = min(int(round(1.2**attempts)), 30)
                print "Connection Failed! Try again in " + str(pause) + " seconds."
                sleep(pause)
                attempts += 1

        self.__loop__()

    def __loop__(self):
        if self.stopped:
            return
        while True and not self.stopped:
            try:
                data = self.socket.recv(1024)
            except:
                try:
                    self.socket.close()
                except:
                    pass
                break
            if not data:
                break
        self.__connect__()

    def send(self, msg):
        try:
            self.socket.send(msg)
            return True
        except:
            self.queue.append(msg)
            return False


m_socket = MSocket()

m_socket will then be imported by the main urls.py so that it starts with django.

So my setup looks kind this:

Sending to TCP Server:

Django (connect:8001) ------->  (listen:8001) Twisted (connect:4444) ------> (listen:4444) TCP-Server

Receiving from TCP Server

TCP-Server (listen:4444) ------> (connect:4444) Twisted ---(direct access)---> Django

It all seems to work that way, but I fear that this is not a really good solution, since I have to open this extra TCP connection. So my question would be now, if the setup can be optimized (and I'm sure it can) and how it can be done.

Answer 1

This is not going to work unless you monkey patch Django (as mentioned by @pss) I had a similar situation so this is what I did.

1. Run a separate twisted deamon.
2. To communicate from Django to Twisted, use Unix sockets. The local twisted server can listen on Unix sockets (AF_UNIX) and Django can simply connect to that socket. This will avoid going through the TCP stack
3. To communicate from Twisted to Django, you have multiple options, a) call Django url with the data b) launch a script (Django management command) c) Use celery to launch the the above Django command d) Use a queue (zeromq or rabbit) and have your Django management command listen in on the queue (preferred)

With the last option, you get much better throughput, durability and it scales well.