Search code examples
flaskherokurediscelery

Celery stuck in a loop - why?

I'm new to celery and redis and I'm trying to understand why celery sometimes get stuck in a loop and keeps creating tasks for a tiny job. Please see logs. I'm not sure if this is normal?

enter image description here

I'm using:

  • Heroku
  • flask==1.1.2
  • redis==4.5.5
  • celery==4.4.2

__init__.py

    #!venv/bin/python3
    # -*- coding:utf-8 -*-
    import os
    import socket
    import eventlet
    from eventlet.green import ssl
    
    
    def make_celery(app_name=__name__):
        eventlet.monkey_patch(all=False, socket=True)
        ssl.timeout_exc = socket.timeout
    
        redis_url = os.environ.get(
            'REDIS_URL', 'redis://localhost:6379/0')
        celery_broker_url = redis_url
        celery_result_backend = redis_url
        celery = Celery(
            app_name, backend=celery_result_backend, broker=celery_broker_url
        )
        return celery
    try:
        from celery import Celery
        celery = make_celery()
    except ImportError:
        celery = None

celery workers

celery worker --app=mycompany.pptxcelery.celery_worker.celery --autoscale=5,1 --max-tasks-per-child=5 --time-limit=300
Solution

  • Leaving our solution here in case it helps anyone in the future.

    The reason for our problem was we were using Redis=4.5.5 which celery specifically says not to use because its buggy.

    Upgrading to 5.0.0 solved the problem.