Search code examples
pythonipctwistedrabbitmqzeromq

Python IPC - Twisted, RabbitMQ,

I want to create 2 applications in Python which should communicate with each other. One of these application should behave like a server and the second should be the GUI of a client. They could be run on the same system(on the same machine) or remotely and on different devices.

I want to ask you, which technology should I use - an AMQP messaging (like RabbitMQ), Twisted like server (or Tornado) or ZeroMQ and connect applications to it. In the future I would like to have some kind of authentication etc.

I have read really lot of questions and articles (like this one: Why do we need to use rabbitmq), and a lot of people are telling "rabbitmq and twisted are different". I know they are. I really love to know the differences and why one of these solutions will be superior than the other in this case.

EDIT: I want to use it with following requirements:

  1. There will be more than 1 user connected at a time - I think there will be 1 - 10 users connected to the same program and they would work collaboratively
  2. The data send are "messages" telling what user did - something like remote calls (but don't focus on that, because the GUIS can be written in different languages, so the messages will be something like json informations).
  3. The system should allow for collaborative work - so it should be as interactive as possible. (data will be send all the time when user something types or performs some action).

Additional I would love to hear why one solution would be better than the other not only in this particular case.

Solution

  • Twisted is used to solve the C10k networking problem by giving you asynchronous networking through the Reactor Pattern. Its also convenient because it provides a nice concurrency abstraction as threading/concurrency in Python is not as easy as say Erlang. Consequently some people use Twisted to dispatch work tasks but thats not what it is designed for.

    RabbitMQ is based on the message queue pattern. Its all about reliable message passing and is not about networking. I stress the reliable part as there are many different asynchronous networking frameworks (Vert.x for example) that provide message passing (also known as pub/sub).

    More often than not most people combine the two patterns together to create a "message bus" that will work with a variety of networking needs with out unnecessary network blocking and for great integration and scalability.

    The reason a "message queue" goes so well with a networking "reactor loop" is that you should not block on the reactor loop so you have to dispatch blocking work to some other process (thread, lwp, separate machine process, queue, etc..). In practice the cleanest way to do this is distributed message passing.

    Based on your requirements it sounds like you should use asynchronous networking if you want the results to show up instantly and scale but you could probably get away with a simple system that just polls given you only have handful of clients. So the question is how many total users (Twisted)? And how reliable do you want the updates to be delivered (RabbitMQ)? Finally do you want your architecture to be language and platform agnostic... maybe you want to use Node.js later (focus on the message queue instead of async networking... ie RabbitMQ). Personally I would take a look at Vert.x which allows you to write in Python.