Beanstalkd support for multiple servers and load balancing

Question

I am using beanstalkd in a Laravel project to handle jobs on a queue. Beanstalkd is running locally. What I want to do is add one or more remote servers to handle some jobs when the queue gets bigger. I know that with Laravel I can send a job to a specific remote connection but in this way I don't know the load in each server prior to sending the job.

I was wondering if beanstalkd supports load balancing between servers and error handling when a remote job fails for example.

Thank you

Answer 1

Beanstalkd does't have features for load balancing.

You could setup a HAProxy on your balancer and signup multiple servers with beanstalkd installed. Then when you send jobs from Laravel code you send to the HAProxy, and HAProxy decides on which sub-server puts the job, as it knows the loading and if there is an incident with a sub system.

In the code you just need to change the IP. In your infrastructure you need to have balancer (HAProxy) that is setup with a pool of Beanstalkd servers.

We usually have 2 machines, and they are configured like this:

- Machine 1: HAProxy, Apache, MySQL, Laravel, Beanstalkd
- Machine 2: MySQL, Laravel, Beanstalkd