Celery Beat unable to find database models (django.db.utils.OperationalError)

Question

I have to add few periodic tasks. I'm using Celery - Redis in Django platform. When I execute the method from shell_plus all is well . However Celery Beat is unable to find the database instance properly.

Celery version = 4.1.0. I had previously installed django-celery-beats etc Database = MySQL

Where am i wrong.

Thanks in advance. Celery Command

(venv)$:/data/project/(sesh/dev)$ celery -A freightquotes worker -B -E -l INFO --autoscale=2,1

settings.py

CELERY_BROKER_URL = 'redis://127.0.0.1:6379'
CELERY_BROKER_TRANSPORT = 'redis'
CELERY_BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 604800}

CELERY_RESULT_BACKEND = BROKER_URL

CELERY_TASK_RESULT_EXPIRES = datetime.timedelta(days=1)  # Take note of the CleanUp task in middleware/tasks.py
CELERY_MAX_CACHED_RESULTS = 1000
CELERYBEAT_SCHEDULER = "djcelery.schedulers.DatabaseScheduler"
CELERY_TRACK_STARTED = True
CELERY_SEND_EVENTS = True
CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']

REDIS_CONNECT_RETRY = True
REDIS_DB = 0
BROKER_POOL_LIMIT = 2
CELERYD_CONCURRENCY = 1
CELERYD_TASK_TIME_LIMIT = 600

CELERY_BEAT_SCHEDULE = {
    'test': {
            'task': 'loads.tasks.test',
            'schedule':  crontab(minute='*/1'),
    },

init.py

from __future__ import absolute_import, unicode_literals

from .celery import app as celery_app

__all__ = ['celery_app']

celery.py

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings.base')

app = Celery('project')

app.config_from_object('django.conf:settings', namespace='CELERY')

app.autodiscover_tasks()


@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))

loads/tasks.py

@task()
def test():
    x = [i.id for i in Load.objects.all()]
    print (x)

Error

[2017-11-30 03:52:00,032: ERROR/ForkPoolWorker-2] Task loads.tasks.test[0020e4ae-5e52-49d8-863f-e51c2acfd7a7] raised unexpected: OperationalError('no such table: loads_load',)
Traceback (most recent call last):

File "/data/project/venv/lib/python3.4/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/data/project/venv/lib/python3.4/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute
    return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: no such table: loads_load

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/project/venv/lib/python3.4/site-packages/celery/app/trace.py", line 374, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/data/project/venv/lib/python3.4/site-packages/celery/app/trace.py", line 629, in __protected_call__
    return self.run(*args, **kwargs)
  File "/data/project/loads/tasks.py", line 146, in test
    x = [i.id for i in Load.objects.all()]
  File "/data/project/venv/lib/python3.4/site-packages/django/db/models/query.py", line 250, in __iter__
    self._fetch_all()
  File "/data/project/venv/lib/python3.4/site-packages/django/db/models/query.py", line 1103, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/data/project/venv/lib/python3.4/site-packages/django/db/models/query.py", line 53, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
  File "/data/project/venv/lib/python3.4/site-packages/django/db/models/sql/compiler.py", line 886, in execute_sql
    raise original_exception
  File "/data/project/venv/lib/python3.4/site-packages/django/db/models/sql/compiler.py", line 876, in execute_sql
    cursor.execute(sql, params)
  File "/data/project/venv/lib/python3.4/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/data/project/venv/lib/python3.4/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/data/project/venv/lib/python3.4/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/data/project/venv/lib/python3.4/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/data/project/venv/lib/python3.4/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: no such table: loads_load

Answer 1

I found some answer.

We have few files in settings like base, dev, prod and local. The database settings is different in each one of them.

Its working when i point the celery app to the local which has all database config. In this case i had to copy all celery config from base to local.

I tried to use django.conf.settings as os.environ.setdefault , but didnt work.

So the answer is incorrect configuration. if we have all in one file we are fine. if we split we have to find some work around.

Edit

Since the issue was finding the right settings file. I now start the celery by setting the module DJANGO_SETTINGS_MODULE='project.settings.dev' celery -A project worker -B -E -l INFO --autoscale=2,1