Celery Django Daemon Multiple Virtual Environment

Question

My problem is the following:

I have a django app on which I can upload files. When a file is uploaded, a celery task is launched to process the file on a specific queue depending of its version like the following:

import my_library

@app.task()
def process_file(file):
    result = my_library.process(file)
    model = MyModel(result=result)
    model.save()
    return

def file_upload(request):
    file = request.FILE['file']
    version = parse_version(file)
    process_file.run_async(file, queue=version)

So I have a queue for each version of my library. My idea was to create multiple daemon, one for each queue/version of my library using virtual environments to use the correct version of my_library.

But I don't know how to correctly do that.

my_library with different versions cannot coexist because they contains Cython functions with dependency on a custom C library with the same version number.

Answer 1

• create a separate virtualenv for each version of the library you need to run
• install the appropriate library inside each virtualenv
• install your code in the virtualenv
• use supervisord to run a separate celery worker for each version/virtualenv. Each worker listens to a different queue
• dipatch the task to the appropriate queue

/etc/supervisor.d/celery.conf

[program:celery-version-1]
command = /path/to_venv_version_1/bin/python manage.py celery worker -Q version-1 ....

[program:celery-version-2]
command = /path/to_venv_version_2/bin/python manage.py celery worker -Q version-2 ....

[program:celery-version-3]
command = /path/to_venv_version_3/bin/python manage.py celery worker -Q version-3 ....

then dispatch your tasks:

def file_upload(request):
    file = request.FILE['file']
    version = parse_version(file)
    process_file.run_async(file, queue=version)