Search code examples
djangocelerydjango-celery

Django-celery on multiple computers

I got everything I wanted to do with django-celery working on my development machine. More specifically, the app accepts photo urls which are then turned into tasks that the same machine downloads.

Now what I want to do is put the django code on heroku and the celery tasks on a dedicated computer that will be kept in the office.

I don't know what the next step is though. How do I tell the django app to connect to the office computer? What is the process for setting up the office computer to accepts tasks from the django app? How do I give the local computer login credentials to the django app so that it can connect to the database to update the models?

Ideally, I am looking to put something like this in my setting.py file:

remote_worker = '123.2.4.23:1234'

and on the office computer 

tasks = 'photos/tasks.py'
remote_app = 'herokuapp123.com/myapp'
username = 'me'
password = 'pw'

I know there are a lot of questions. Any help or pointers would be appreciated!

Solution

  • This largely depends on what AMQP backend you are using for celery. If you are using the default (RabbitMQ) you will need do one of the following:

    1. Install RabbitMQ on heroku server, expose its port to your business IP through firewall and configure your office computer to connect to it
    2. Install RabbitMQ locally on your business computer and configure celery on Heroku to connect to it
    3. Install RabbitMQ on both sides and bridge them.

    Alternatively you can integrate the heroku server in your own business network using a VPN solution and have them directly talk to each other (because after all you probably don't want to transmit AMQP packets bare over the interwebz).

    Scenario 1 is probably the easiest to set up as Heroku already provides you the plugin infrastructure to do so. Scenario 2 is probably not what you want as you will have to punch a hole in your business firewall for that. Both scenarios 1 and 2 will have latency and reliability issues as routing AMQP traffic over the internet is not going to be expedient or reliable. You will have dropped messages and celery will keep retrying until it succeeds or reaches the max number of failures. However AMQP was designed to handle network issues, they just may inadvertently affect your performance if that is critical. But then again in that case you should reconsider putting the celery workers on a business desktop.

    Scenario C is probably best in terms of reliability but also more difficult to set up. Choose based on your needs.