Search code examples
celeryamazon-elastic-beanstalksupervisord

Limit number of processes in Celery with supervisor

I'm running Celery in a small instance in AWS Elastic Beanstalk.

However, when I do top, I see there are 3 celery processes running. I want to have only.

I'm running this using supervisor and in my config file I have (only showing relevant lines):

[program:celeryd]
directory=/opt/python/current/app/src
command=/opt/python/run/venv/bin/celery worker -A ..."

user=celery
numprocs=1
killasgroup=true

I've also followed the suggestion in this answer and created a file /etc/default/celeryd with this content:

# Extra arguments to celeryd
CELERYD_OPTS="--concurrency=1"

After restarting Celery (with supervisorctl -c config-file-path.conf restart celeryd), I see the 3 processes again. Any ideas? Thanks!

Solution

  • You are starting worker with celery command. Changing /etc/default/celeryd won't have any effect on celery command. Moreover celeryd is deprecated.

    When a worker is started, celery launches a default process and n(concurrency) subprocesses.

    You can start the worker with 

    [program:celery]
command=/opt/python/run/venv/bin/celery worker -c 1 -A foo"

    This will start a worker with concurrency of 1 and there will be 2 processes.