Running rqworker concurrently

Question

I'm new to RQ and am trying to use it for a job which will run in the background. I have managed to set it up, and I'm also able to start more than one worker.

Now I'm trying to run these workers concurrently. I installed supervisor and followed a tutorial to add programs to it, and it worked.

Here is my supervisor configuration:

[program:rqworker]

command=/usr/local/bin/rq worker mysql

process_name=rqworker1-%(process_num)s

numprocs=3

directory=/home/hp/Python/sample

stopsignal=TERM

autostart=true

autorestart=true

stdout_logfile=/home/hp/Python/sample/logs

The worker function is present in the sample directory mentioned above. The problem is that even after specifying numprocs as 3 in the config file, the workers do not run in parallel.

Here are some screenshots, which show that although multiple workers have been started, they do not work in parallel.

Also, I saw this stackoverflow answer, but it still doesn't divide the jobs amongst the workers! Could anyone tell me what is wrong with this configuration/what I need to change?

Answer 1

I found the problem; it wasn't with supervisor or rqworker. The manager program was blocking concurrency, by waiting for task completion!