Celery Task State Always Pending in Django

Question

I'm having issues with my Celery tasks in Django. When I dispatch a task, it remains in the PENDING state forever, even though it seems like it's being received by the worker. I'm using Redis as the broker and django-db as the result backend.

Here are the relevant parts of my settings.py:

CELERY_BROKER_URL = "redis://localhost:6379"
CELERY_RESULT_BACKEND = "django-db"
CELERY_TASK_TRACK_STARTED = True

I have defined the task in my tasks.py:

from celery import shared_task
from channels.layers import get_channel_layer
from asgiref.sync import async_to_sync
import subprocess
import time
from celery.utils.log import get_task_logger

logger = get_task_logger(__name__)

@shared_task
def run_image_processing_script():
    try:
        channel_layer = get_channel_layer()

        process = subprocess.Popen(
            ["python", "BLACKJACK_CARDS_COUNTING_PROJECT_2.py"], stdout=subprocess.PIPE
        )
        ...
        (rest of the function)
        ...
    except Exception as e:
        logger.exception("Error in run_image_processing_script: %s", e)
        raise

I am running the worker and it seems to be receiving the tasks:

celery@MrRobot ready.
Task screenshot.tasks.run_image_processing_script[120f4cd1-ee5a-4bff-a217-f59e7e074ab4] received

However, when I check the task state, it's always PENDING.

>>> from screenshot.tasks import run_image_processing_script
>>> result = run_image_processing_script.delay()
>>> print(result.state)
PENDING

I've already tried setting CELERY_TASK_TRACK_STARTED = True in my settings.py and it did not help. I have also verified that there are no exceptions in the Celery worker logs, and the external script BLACKJACK_CARDS_COUNTING_PROJECT_2.py runs correctly on its own.

Any help would be greatly appreciated.

Answer 1

Celery stores the task status in the backend. As you are using database, you can check in the table the task statuses. By default celery, shows PENDING state for any task_id, if it can retrieve it's status.

Celery is not generally used to run a task in the background for indefinite amount of time. You should consider other ways of running it. It's mostly used to offload time-taking tasks to different processes(worker) running on same/different machines. For debugging, you can run the worker in DEBUG mode by setting --loglevel=DEBUG on the CLI. If you just want to send the current status of this long running job, you are better off with manually updating the status in a database at defined points and send that information through django channels. You can use process control library like supervisord to manage the process, if you are running things directly on the server.