Search code examples
djangoceleryvirtualenvproduction-environment

Django celery WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL) error

What is the best practice to run celery as a daemon in a production virtualenv? I use the following in the local environment which works perfect and receiving tasks works as expected. But in production always stuck at 

WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL) error

I use the following configuration in local and in production:

/etc/default/celeryd:

CELERY_BIN="path/to/celery/bin"
CELERY_APP="myproj"
CELERYD_CHDIR="home/myuser/project/myproj"
CELERYD_OPTS="--time-limit=300 --concurrency=4"
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERYD_USER="myuser"
CELERYD_GROUP="myuser"
CELERY_CREATE_DIRS=1

/etc/init.d/celeryd:[celeryd]

Package & OS version info:

  1. Ubuntu == 16.04.2
  2. Celery == 4.1.0
  3. rabbitmq == 3.5.7
  4. django == 2.0.1

I also use these commands while making celery to run as daemon:

  1. sudo chown -R root:root /var/log/celery/
  2. sudo chown -R root:root /var/run/celery/
  3. sudo update-rc.d celeryd defaults
  4. sudo update-rc.d celeryd enable
  5. sudo /etc/init.d/celeryd start

Here is my django settings.py configuration for celery:

CELERY_BROKER_URL = 'amqp://localhost'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_RESULT_BACKEND = 'db+sqlite:///results.sqlite'
CELERY_TASK_SERIALIZER = 'json'

Need expert advise to make the celery daemon to work correctly in production virtualenv. Thanks in advance!

Solution

  • I got this error due to out of memory error caught in

    /var/log/kern.log

    I have Tensorflow running in one of my tasks, which needs additional computational power, but my physical memory(RAM) is not sufficient to handle that much load. I weird that there's no log in celery except SigKill 9 error. But the kernel log helped me to fix it.