I have a two simple periodic celery tasks which run at night, one at 2AM, other at 3AM, and from time time throw following exception about a deadlock on djcelery_periodictask table which I do not quite understand. It seems that one task tries to update djcelery_periodictask while other is still holding a share lock on it:
deadlock detected Detail: Process 32353 waits for ShareLock on transaction 39280165; blocked by process 32306. Process 32306 waits for ShareLock on transaction 39280166; blocked by process 32353.
Process 32353:
UPDATE "djcelery_periodictask" SET "name" = 'app.tasks.task1', "task" = 'app.tasks.task1', "interval_id" = NULL, "crontab_id" = 4, "args" = '[]', "kwargs" = '{}', "queue" = NULL, "exchange" = NULL, "routing_key" = NULL, "expires" = NULL, "enabled" = true, "last_run_at" = '2015-02-10 03:00:00.001297+01:00', "total_run_count" = 117, "date_changed" = '2015-02-10 02:00:00.010033+00:00', "description" = '' WHERE "djcelery_periodictask"."id" = 4
Process 32306:
UPDATE "djcelery_periodictask" SET "name" = 'app.tasks.task2', "task" = 'app.tasks.task2', "interval_id" = NULL, "crontab_id" = 6, "args" = '[]', "kwargs" = '{}', "queue" = NULL, "exchange" = NULL, "routing_key" = NULL, "expires" = NULL, "enabled" = true, "last_run_at" = '2015-02-10 02:00:00.014189+01:00', "total_run_count" = 63, "date_changed" = '2015-02-10 02:00:00.015097+00:00', "description" = '' WHERE "djcelery_periodictask"."id" = 6
Hint: See server log for query details. Query:
UPDATE "djcelery_periodictask" SET "name" = 'app.tasks.task2', "task" = 'appt.tasks.task2', "interval_id" = NULL, "crontab_id" = 4, "args" = '[]', "kwargs" = '{}', "queue" = NULL, "exchange" = NULL, "routing_key" = NULL, "expires" = NULL, "enabled" = true, "last_run_at"
How can be that two a simple updates on a table containing just a few rows come to a deadlock?
My problem was that I had all of my workers started with -B parameter which turned each worker in a periodic task scheduler:
-B, --beat Also run the celery beat periodic task scheduler.
Please note that there must only be one instance of
this service.
As a result, the scheduled tasks were started multiple times, all at the same time, creating a deadlock when trying to update djcelery_periodictask table.