Search code examples
djangocelerydjango-celerycron-taskcelerybeat

Celery and Celerybeat randomly start acting funky

I am using Celery with my Django app and for the most part, things are running grand. For reference, here is my celerybeat configruation:

CELERYBEAT_SCHEDULE = {
    'update-every-5-secs': {
        'task': 'app1.tasks.task1',
        'schedule': timedelta(seconds=5),
        'args': (),
    },
    'archive-request-records-every-hour': {
        'task': 'app0.tasks.archive_requests',
        'schedule': crontab(minute=0, hour='*/1'),
        'args': (),
    },
    'push-ios-background-every-5-minutes': {
        'task': 'app0.tasks.push_ios_background',
        'schedule': crontab(minute='*/5'),
        'args': (),
    },
}

I've noticed that every 1-2 days, it will just randomly fall completely out of sync. For example, the last entry that runs every 5 minutes normally runs like so: 1:00, 1:05, 1:10, 1:15, etc.

Here is what it did this morning: 7:30, 7:35, 7:40, 7:46, 7:50, 8:03, 8:16, 8:32, 8:50, 9:13, 9:21, 9:26, 9:29, 9:32, 9:38, 9:47, 9:59, 10:08, 10:22, 10:24, 10:24, 10:24, 10:24, 10:25, 10:25, 10:25, 10:25, 10:25, 10:25, 10:25, 10:25, 10:25, 10:25, 10:25, 10:26, 10:26, 10:30, 10:35, 10:40...

It almost looks like it falls way out of sync, attempts to make up for it all of once, and then regains itself. At first I thought it was just a fluke, but it is happening way too frequently. I have mission critical processes on this that need to run on time.

Is there any way to prevent this from happening? It seems like people have had tons of success with Celery so I am truly baffled.

Solution

  • Your settings look good.

    Could it be that you are measuring when the task is completed, not when it is starting? It could be that your tasks take very long or even fail so they do not get reported.

    crontab is low level and fairly accurate in my experience, so I'd check your logging tool first.

    Update

    Another debugging solution is to start implementing time limits for tasks to make sure they do not get locked too long. http://docs.celeryproject.org/en/latest/userguide/workers.html#time-limits

    CELERYD_TASK_TIME_LIMIT = 300
CELERYD_TASK_SOFT_TIME_LIMIT = 260