Django Celery Directory Structure and Layout

Question

I have a django project using the following directory structure.

project/
  account/
      models.py
      views.py 
  blog/
      models.py
      views.py
  mediakit/
      models.py
      views.py
  reports/
      celery.py <-- new
      models.py
      tasks.py <-- new
      views.py
  settings/
      __init__.py <-- project settings file
  system/
      cron/
      mongodb/
      redis/
  manage.py

Here's the contents of celery.py derived from the celery tutorial (http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html).

from __future__ import absolute_import

import os

from celery import Celery

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'settings')

from django.conf import settings

# app = Celery('reports')
app = Celery('reports',
             backend='djcelery.backends.database:DatabaseBackend',
             broker='amqp://guest:guest@localhost:5672//')

# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)


@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))

Some of my apps are shared across projects. reports, for example might be used in 4 different projects, so I can see how tasks.py should live in the reports app so when it's added to a new project the tasks come along. What I don't quite understand is why celery.py needs to live within the reports app too. When I go to add some tasks to the account app, I'm basically building the same celery.py file replacing 'reports' with 'account'. Shouldn't I have one celery file that lives at the same level as manage.py? Any help or suggestions would be greatly appreciated.

Answer 1

The celery app file should live in the core directory of your project, along the settings and all the other things as shown in the documentation that you posted.

To define portable tasks it makes sense to put them in the app that is using them, as you pointed out, in your case the reports app.

The idea is that your task file is registered by whatever celery app is defined in the project, and your django app need no knowledge of which celery app is registering the tasks. You do this by using the shared_task decorator instead of app.task.

To summarize:

project/
  project/
      settings.py
      celery.py <- new, shown in the docs, also add __init__.py
      urls.py
  account/
      models.py
      views.py 
  blog/
      models.py
      views.py
  mediakit/
      models.py
      tasks.py <-- tasks for the me
      views.py
  reports/
      models.py
      tasks.py <-- tasks for the reports app
      views.py
  manage.py

In tasks.py you have something like this:

from celery import shared_task

@shared_task
def my_add_task(a, b):
    return a + b

Hope this helps.