Search code examples
djangoceleryamazon-ecscelerybeat

Setup celery worker and celery beat (django app) on AWS ECS

I have a Django app with celery worker and beat, and I'm having a hard time trying to set up celery on ECS (The configuration of the Django app on ECS was pretty straightforward, no problems).

1) What kind of service should I use for the worker and the beat: REPLICA or DAEMON?

service

2) What command could I use to the Healthcheck on the task definition configuration?

task_definition

I tried using supervisor and also simply calling the celery command directly in the command section of the task definition.

I don't know what exactly I am missing...

  • Supervisor configuration files:

supervisor.conf:

[unix_http_server]
file=/var/run/supervisor.sock   ; (the path to the socket file)
chmod=0700                      ; sockef file mode (default 0700)

[supervisord]
logfile=/var/log/supervisor/supervisord.log ; supervisord log file
logfile_maxbytes=50MB       ; maximum size of logfile before rotation
logfile_backups=10          ; number of backed up logfiles
loglevel=debug               ; info, debug, warn, trace
pidfile=/var/run/supervisord.pid ; pidfile location
nodaemon=false              ; run supervisord as a daemon
minfds=1024                 ; number of startup file descriptors
minprocs=200                ; number of process descriptors
user=root                   ; default user
childlogdir=/var/log/supervisor/            ; where child log files will live

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL  for a unix socket
[include]
files = /etc/supervisor/conf.d/*.conf

celeryworker.conf:

[program:celery]
command=/start-worker.sh

directory=/app
user=myuser
numprocs=1
stdout_logfile=/var/log/app_logs/celery-worker.log
stderr_logfile=/var/log/app_logs/celery-worker.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs = 600

stopasgroup=true

priority=1000

celerybeat.conf:

[program:celerybeat]
command=/start-beat.sh

directory=/app
user=myuser
numprocs=1
stdout_logfile=/var/log/app_logs/celery-beat.log
stderr_logfile=/var/log/app_logs/celery-beat.log
autostart=true
autorestart=true
startsecs=10
process group.

stopasgroup=true

priority=999
  • Running scripts:

start-worker.sh:

#!/usr/bin/env bash
celery -A my_app worker -l ${APP_LOG_LEVEL} -n worker.%%h

start-beat.sh:

#!/usr/bin/env bash
celery -A my_app beat -l ${APP_LOG_LEVEL} --workdir=/app --schedule django
Solution

  • Just created the Service with a Minimal healthy percentage = 0 and the Task Definition without healthcheck as pointed out by @ItayB. I also removed the supervisor setup.