Celery task state always pending

Question

I am pretty new to celery and django in general so please excuse my lack of knowledge. I am trying to run a test to do some calculations and wait for the test to finish so that I can make sure it is done for the correct answers.

Here is what i have:

In app/tests.py

from tasks import *


c = calculate.apply_async(args=[1])

# wait until the task is done
while not calculate.AsyncResult(c.id).status == "SUCCESS":
    print c.state
    pass

in app/tasks.py

from celery import shared_task

@shared_task
def calculate(proj_id):

    #some calculations followed by a save of the object

The state never changes from pending even though in the celery log it says that the task was completed successfully

[2014-06-10 17:55:11,417: INFO/MainProcess] Received task: app.tasks.calculate[1f11e7ab-0add-42df-beac-3d94c6868aac]
[2014-06-10 17:55:11,505: INFO/MainProcess] Task app.tasks.calculate[1f11e7ab-0add-42df-beac-3d94c6868aac] succeeded in 0.0864518239978s: None

I have also put CELERY_IGNORE_RESULT = False in the mainapp/settings.py, but this did not seem to do anything.

Answer 1

So, your settings are wrong. :) You also need to setup a broker for celery to work.

First of all, djcelery is deprecated, everything is included in celery for it to work with django.

Second of all, do not set all content to be accepted, it can be a potential security risk. Use pickle only in case that simple json is not enough (let's say you pass functions or objects as arguments to tasks, or return from tasks)

So my guess is, you are just trying celery out, that's why you are trying to use database backend, which is fine, but for production use I would recommend using RabbitMQ.

In any case, give it a try with those settings:

BROKER_URL = 'django://'
INSTALLED_APPS = (
    ...
    'kombu.transport.django', 
    ...
)
CELERY_RESULT_BACKEND = 'db+scheme://user:password@host:port/dbname' 
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_IGNORE_RESULT = False # this is less important

then run python manage.py syncdb

Just to let you know, I have not used database as a broker or result backend, hence setup might be incomplete, or even incorrect, but give it a try anyways.

more CELERY_RESULT_BACKEND setting for database examples

In case you want to setup RabbitMQ as a broker backend, which I would recommend and I know for sure it will work:

if on ubuntu run:

sudo apt-get install rabbitmq-server
sudo rabbitmqctl add_user <username> <password>
sudo rabbitmqctl add_vhost <vhost, use project name for example>
sudo rabbitmqctl set_permissions -p <vhost> <username"> ".*" ".*" ".*"

Then configure celery in settings.py:

BROKER_URL = 'amqp://<user>:<password>@localhost:5672/<vhost>'
CELERY_TIMEZONE = TIME_ZONE
CELERY_RESULT_BACKEND = 'amqp'
# thats where celery will store scheduled tasks in case you restart the broker:
CELERYD_STATE_DB = "/full/path/data/celery_worker_state" 
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'

Let me know how it goes.