Search code examples
pythondjangoredisrabbitmqcelery

Celery: When should you choose Redis as a message broker over RabbitMQ?

My rough understanding is that Redis is better if you need the in-memory key-value store feature, however I am not sure how that has anything to do with distributing tasks?

Does that mean we should use Redis as a message broker IF we are already using it for something else?

Solution

  • I've used both recently (2017-2018), and they are both super stable with Celery 4. So your choice can be based on the details of your hosting setup.

    • If you must use Celery version 2 or version 3, go with RabbitMQ. Otherwise...
    • If you are using Redis for any other reason, go with Redis
    • If you are hosting at AWS, go with Redis so that you can use a managed Redis as service
    • If you hate complicated installs, go with Redis
    • If you already have RabbitMQ installed, stay with RabbitMQ

    In the past, I would have recommended RabbitMQ because it was more stable and easier to setup with Celery than Redis, but I don't believe that's true any more.

    Update 2019

    AWS now has a managed service that is equivalent to RabbitMQ called Amazon MQ, which could reduce the headache of running this as a service in production. Please comment below if you have any experience with this and celery.

    Update 2025

    The Celery Using Redis documentation lists some caveats for choosing Redis, which includes limitations with

    • Visibility timeout - If a task isn’t acknowledged within the Visibility Timeout the task will be redelivered to another worker and executed.
    • Key eviction - In some circumstances a key could be removed unexpectedly.
    • Group result ordering - For older versions of celery, results could be returned in an unexpected order.