How to deploy tornado websocket multiprocess application with supervisord?

Question

I have an application (small part, unnecessary is skipped) :

# -*- coding: utf-8 -*-
import tornado.web
import tornado.httpserver
import tornado.ioloop
import tornado.websocket
import db_util # pg utility import

class SampleWebSocket(tornado.websocket.WebSocketHandler):

    def open(self, *args, **kwargs):
        print u"open Websocket"
        do_init_stuff()

    def on_message(self, data):
        print u"%s BASE DATA RECIEVED:"
        print data
        do_on_message_stuff()

    def on_close(self):
        print u"WebSocket closed"
        do_on_close_stuff()

    def check_origin(self, origin):
        # Allow connect from all domains
        return True

def make_app(_settings):
    routes = [
        (r"/chat/sample/", handlers.SampleWebSocket),
    ]
    app = tornado.web.Application(routes)
    app.db = db_util.ConnectionRegistry()
    return app


if __name__ == "__main__":
    app = make_app()
    port = 5001
    app.db = db_util.ConnectionRegistry()
    print " *** DEBUG:False, Starting Chat at port: %s" % port
    server = tornado.httpserver.HTTPServer(app)
    server.bind(port)
    server.start(0) # actual number of cores == 2, so 2 subprocesses will be born
    tornado.ioloop.IOLoop.instance().start()

I am trying to deploy it with supervisord. Here is part of supervisord config:

[program:chat]
directory=/home/me/chat
command=/home/me/venv/bin/python2.6 /home/me/chat/main.py
process_name=%(program_name)s_%(process_num)01d
redirect_stderr=true
stdout_logfile=/tmp/chat.out.log
numprocs=1
numprocs_start=5001

When I start it at first time with $ supervisord -c supervisord.conf it starts, responds, but seems like supervisord does not control tornado child processes, so if I restart service with $ supervisorctl restart, I still have tornado childs living, binded to 5001 port. And service is unable to restart correctly.

So, does exists some way to configure supervisord for tornado with multiprocessing?

Answer 1

In general, if you're using supervisord you should be using supervisord's process groups instead of tornado's multi-process mode. Increase numprocs in the supervisor config and give each process a different port by passing --port=%(process_num)s in the command line (with a suitable option defined in the app itself). You'll need a proxy like nginx or haproxy to balance among the processes (or you could use file-descriptor passing to share one port as seen in https://gist.github.com/bdarnell/1073945, but this is an advanced usage and I recommend using a proxy instead).