Search code examples
pythondjangocelery

Celery task stays running forever, but completes successfully


My Task:

@app.task(ignore_result=True)
def leaks_scan(target: str, remove_type: str = "old", **kwargs):
    url = settings.API_URL + "leaks"
    skip = 0
    limit = 1000
    max_limit = 5000
    if remove_type == "all":
        remove_all_data(Leak, target)

    scan_id = None
    target_id = None
    while True:
        data = get_data(url, target, skip=skip, limit=limit)
        count = data.get("count")
        data = data.get("data", [])
        if not data:
            if not count:
                remove_all_data(Leak, target)
                break
            break
        scan_id, target_id = load_leak_data(data)
        skip += limit
        if skip > max_limit:
            break

    if remove_type == "old" and scan_id and target_id:
        remove_old_data(Leak, scan_id, target_id)
    print("Leak scan completed")

Notice that this task has a print statement that prints Leak scan completed in the last line

Logs:

Task received:

resim

Task finished (debug print line):

resim

The task works as expected and completes successfully. However, when I check the Celery worker logs, the task stays in the running state indefinitely, even though it has been completed:

resim

Can anyone suggest a solution or provide any guidance on how to troubleshoot this issue?

Thanks in advance for your help.

Environment & Settings

Celery version:

celery report Output:

software -> celery:5.2.7 (dawn-chorus) kombu:5.2.4 py:3.11.2
            billiard:3.6.4.0 py-amqp:5.1.1
platform -> system:Linux arch:64bit, ELF
            kernel version:5.15.90.1-microsoft-standard-WSL2 imp:CPython
loader   -> celery.loaders.default.Loader
settings -> transport:amqp results:disabled

deprecated_settings: None

Broker & Result Backend Settings

CELERY_BROKER_URL="redis://localhost:6379/0"
CELERY_RESULT_BACKEND="redis://localhost:6379/0"

Solution

  • It was because of the following part:

    @app.task(ignore_result=True)
    

    I am calling this function using celery's chords function. I missed the part where Celery says:

    celery docs

    I set the ignore_result option to False and now everything works just fine.