I am using celery-django to queue tasks on my site backend. I am trying to create a setup where I have two queues named "low" and "high" and two workers W1 and W2. I want them to consume the tasks from the queue in the following way:
W1 <-- low, high
W2 <-- high
Normally it can be done like this.
Open terminal 1 and enter
$celery worker -n W1 -Q low,high
Open terminal 2 and enter
$celery worker -n W2 -Q high
However I am trying to do the same via celeryd daemon.
I am following the steps given in the link: http://celery.readthedocs.org/en/latest/tutorials/daemonizing.html#example-configuration But the available options don't seem enough to fit the requirement.
Please help me with some configs that I am unaware of which could make it possible. I would prefer not to run multiple daemons or use additional tools like supervisord unless really necessary (Maybe you could advice me on this as well).
You can use CELERYD_OPTS option passing -Q parameter similar to those examples from Celery reference:
# Advanced example starting 10 workers in the background:
# * Three of the workers processes the images and video queue
# * Two of the workers processes the data queue with loglevel DEBUG
# * the rest processes the default' queue.
$ celery multi start 10 -l INFO -Q:1-3 images,video -Q:4,5 data
-Q default -L:4,5 DEBUG
# You can show the commands necessary to start the workers with
# the 'show' command:
$ celery multi show 10 -l INFO -Q:1-3 images,video -Q:4,5 data
-Q default -L:4,5 DEBUG