Search code examples
rabbitmqcelerydjango-celery

Should I have multiple rabbitmq brokers to load balance work?

My application spins up a large number of EC2 machines with celery and rabbitmq installed. Currently I have all my machines acting as workers, with the exception of one machine solely acting as a rabbitmq broker. All my machines' celeryconfig files basically point to that one rabbitmq broker.

I have enabled clustering and HA queues on my configuration, and I specified a certain number of machines to mirror queues in disk. My question is: should I specify multiple rabbitmq brokers instead of just one? Lets say I specify two broker machines -- then half of my celery workers will point to one broker, and the other half will point to another.

I'd prefer not to specify multiple brokers unless having one broker would result in a bottleneck if that one broker gets overworked (is this possible)?

EDIT: The reason I have enabled clustering is to specify another machine as a broker in case the main one fails.

Solution

  • I think it depends on following considerations

    • Throughput and heat point? too many messages pub/consume on queues? or worker over loaded?
    • High Availability and Load Balancing on MQ nodes? tcp load balancer can served as HA and load balancer and node number can be altered at this point rather than all MQ clients, so that all worker will not lost connection when one of MQ node down.