Search code examples
pythonlinuxdjangocelerydaemon

Force celery daemon to use Python 3

So I have struggled with this quite a while but I can't seem to find a solution.

I have installed celery with

pip3 install --user celery

And everything was working fine until I try to run a celery worker and beat as a daemon following the official documentation http://docs.celeryproject.org/en/latest/userguide/daemonizing.html

The specific problem I have is that the configuration file for the daemon needs the full route to celery, which I have located and specified in the file like this

CELERY_BIN="/home/my_home/.local/bin/celery"

but when I run the daemon it says that I do not have a module named Celery

me@my_computer:/etc/default$ sudo /etc/init.d/celeryd start
celery init v10.1.
Using config script: /etc/default/celeryd
Traceback (most recent call last):
  File "/home/my_home/.local/bin/celery", line 7, in <module>
    from celery.__main__ import main
ImportError: No module named 'celery'

Trying different things I think I have found the problem, apparently Celery daemon is trying to run celery using python 2 but since I installed it for python 3 it says that it can't find it. Any clues on how to fix this or if I should try something else?

Additional output:

Python 2:

Python 2.7.6 (default, Oct 26 2016, 20:32:47) 
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from celery.__main__ import main
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named celery.__main__

Python 3:

Python 3.4.3 (default, Nov 17 2016, 01:11:57) 
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from celery.__main__ import main
>>>
Solution

  • You could use venv to containerize the call so that no other python binary is available.

    I run my django environments uniformly containerized with venv, because then you can upgrade one app's dependencies without upgrading all of them. You don't have to containerize the whole app, but then your system environment might get out of sync' with your venv, causing weird issues. It probably would be better to containerize both django and celery in the same venv and then invoke both like

    PATH/TO/VENV/bin/python ...