Search code examples
djangopostgresqlrediscelerytornado

Tornado DB notifications from Django


I have Django server that servers HTTP requests and updates Postgres DB. There is also Tornado TCP server that should push new data from Database.

The problem is: best way to notify Tornado about DB changes. That's it.

What options did I think of:

  1. Celery. I've read a lot of docs and it seems that this is just a queue for tasks, not for communications. Moreover, those communications should be established between different processes.
  2. Redis. It seems like a good solution that also has brukva asynchronous client for Tornado. I'll need to introduce additional dependency on a project though.
  3. Socket. It seems the most lightweight solution, besides the Tornado TCP server serves lots of client sockets anyways, so adding one more shouldn't be a big deal. I don't want to go this low though.
  4. DB notifications. This was a solution I came up at first, but disliked later. Postgres has native notifications support by mechanisms of NOTIFY and LISTEN that are also asynchronous and can be easily added to Tornado IOLoop (io_loop.add_handler with connection file descriptor). This is low level too, but more important is that this solution violates Data Independency Principle.

Now I need to mention a subtle detail that can make a big difference with what I've described:

99% of changes in DB occur not in Django process, but in separate scripts (and hence separate processes) that are launched by scheduler and use Django ORM.

Now this description more or less reflects the actual state of project. So is Celery really not suited for solving this kind of problem? Is Redis the best solution? Are there any other that I don't know about? Any thoughts and suggestions are appreciated.


Solution

  • After evaluating all these options, I decided to go with Redis+Tornado approach. It showed itself quite robust and stable (it was 5 years ago, just for the reference). I laid out the whole project with a working example here: https://www.databrawl.com/2017/04/27/real-time-python-via-tcp/