Search code examples
mongodbconcurrencycelerycelerybeat

How do I coordinate a cluster of celery beat daemons?

I have a cluster of three machines. I want to run celery beat on those. I have a few related questions.

  1. Celery has this notion of a persistent scheduler. As long as my schedule consists only of crontab entries and is statically defined by CELERYBEAT_SCHEDULE, do I need to persist it at all?
  2. If I do, then do I have to ensure this storage is synchronized between all machines of the cluster?
  3. Does djcelery.schedulers.DatabaseScheduler automatically take care of concurrent beat daemons? That is, if I just run three beat daemons with DatabaseScheduler, am I safe from duplicate tasks?
  4. Is there something like DatabaseScheduler but based on MongoDB, without Django ORM? Like Celery’s own MongoDB broker and result backend.
Solution

  • Currently Celery doesn't support multiple concurrent celerybeat instances.

    You have to ensure only a single scheduler is running for a schedule at a time, otherwise you would end up with duplicate tasks. Using a centralized approach means the schedule does not have to be synchronized, and the service can operate without using locks.

    http://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html