Search code examples
phplaravelqueuesupervisordbeanstalkd

Beanstalkd + supervisor + laravel : Queue processing with more than 10000 jobs an hour

I have a project in Larave 5.2 and I am using:

  • Beanstalk
  • Supervisor
  • Laravel 5.2
  • Digital Ocean Hosting with 4GB RAM

The project is mainly based on webhooks. Other website calls our webhook and i add those webhooks in a queue. Roughly, 10000 jobs an hours are being added to the queue.

I have 50 num_process set in supervisor configs.

Can you please suggest me that how can i process the jobs in queue really fast. so that I don't have to wait for hours to get my job processed.

Here is a screenshot of current status in the queue

enter image description here

Any help is highly appreciated.

Thank you

Supervisor Config:

[program:laravel_queue]
command=php /var/www/html/nivesh/artisan --env=production --timeout=3600 queue:listen --queue=important,urgent,high,default
autostart=true
autorestart=true
process_name=%(program_name)s_%(process_num)s
numprocs=55
stderr_logfile=/var/log/laraqueue.err.log
stdout_logfile=/var/log/laraqueue.out.log
priority=999
numprocs_start=55
startsecs=0
redirect_stderr=true
Solution

  • Check if you don't have remote calls to external URLs.

    Also add hints in various places to see which operation takes long.

    Try to break into multiple smaller events all the queue, don't be 1 long tasks, make a chain of events.