Search code examples
djangocelerydjango-celery

Isolate celery for development and production

Im using the same server for both my development server and production server, using virtualenv.

My problem is celery not knowing what project to run the task in. I do not want task from production running on my development server and vice versa.

I have tried to use different broker adress, but it does not work correctly:

supervisor script for production:

[program:production-celery]
command=/home/user/.virtualenvs/production.site.com/bin/celery --app=myproject.celeryconfig:app worker -E -n production --loglevel=INFO --without-mingle --without-gossip -Q default,celery

directory = /home/user/.virtualenvs/production.site.com/myproject
environment=DJANGO_SETTINGS_MODULE='myproject.settings.production'

Development:

[program:development-celery]
command=/home/user/.virtualenvs/development.site.com/bin/celery --app=myproject.celeryconfig:app worker -E -n development --loglevel=INFO --without-mingle --without-gossip -Q default,celery

directory = /home/user/.virtualenvs/development.site.com/myproject
environment=DJANGO_SETTINGS_MODULE='myproject.settings.development'

Production celeryconfig:

import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproject.settings.production')

app = Celery('myproject', broker='amqp://', backend='amqp')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)


app.conf.update(
    CELERY_ACCEPT_CONTENT = ['pickle', 'json'],
    CELERY_TIMEZONE='Europe/Oslo',
    CELERY_ENABLE_UTC=True,
    CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler',
    CELERY_SEND_TASK_ERROR_EMAILS = True,
    CELERY_SEND_ERROR_EMAILS = True,
    CELERY_STORE_ERRORS_EVEN_IF_IGNORED = True,
    CELERY_IGNORE_RESULT = False,
    CELERY_TASK_RESULT_EXPIRES = 172800,
)

Development celeryconfig:

import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproject.settings.development')
app = Celery('myproject', broker='amqp://development:development@localhost/development')
app.config_from_object('django.conf:settings')

app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)


app.conf.update(
    CELERY_ACCEPT_CONTENT = ['pickle', 'json'],
    CELERY_TIMEZONE='Europe/Oslo',
    CELERY_ENABLE_UTC=True,
    CELERY_SEND_TASK_ERROR_EMAILS = False,
    CELERY_STORE_ERRORS_EVEN_IF_IGNORED = True,
    CELERY_IGNORE_RESULT = False,
)

Anyone know why task from production would be run on development?

Solution

  • TL;DR

    Separate your dev and production server as fast as you can.

    Full Answer

    Having both dev and production on the same server can cause a lot of problems.

    You will spend a lot of time by coding code that handle edge cases that might accrue. For instance, dev might have a new feature that you test but it had a bug and you got memory issues - your production might get hurt.

    Another thing is the 3th party services you use, like rabbitMQ - you tried to define different queues but like you saw you had issues and you need to write more code to maintain it(def make_sure_new_feature_not_deleteing_users_on_prudction()). This might happen every time you start playing with something new(redis, memcache, sentry, etc) you will have to configures different ports/urls/queues name/

    Best solution will be to have the exact same code that runs (almost) the same configurations on different machines.