I am trying to run a huey task queue on elastic beanstalk that is needed by my Flask app. But there is no built in way to run huey as a daemon process. The author of huey has advised to run huey with supervisor (this link) and since elastic beanstalk already uses supervisor, I thought we could just add the program to be managed by supervisor. But I am not sure how to do this programatically. Currently, I am using the container_commands (ref link) key in the config file to run this but elastic beanstalk gives me a timeout error after sometime as it runs in the foreground. Below is the config file I am using.
packages:
yum:
gcc: []
gcc-c++: []
gcc-gfortran: []
htop: []
make: []
wget: []
atlas-devel: []
lapack-devel: []
commands:
01enable_swap:
command:
- sudo dd if=/dev/zero of=/var/swap1 bs=1M count=1024
- sudo mkswap /var/swap1
- sudo chmod 644 /var/swap1
- sudo swapon /var/swap1
cwd: /home/ec2-user
02install_redis:
command:
- wget "http://download.redis.io/redis-stable.tar.gz"
- tar -xvzf redis-stable.tar.gz
- rm redis-stable.tar.gz
- cd redis-stable
- sudo make
- sudo make install
cwd: /home/ec2-user
container_commands:
01download_nltk_packages:
command: "python install_resources.py"
02run_redis:
command: "redis-server --host 127.0.0.1 --port 6379 --daemonize yes"
03run_huey:
command: "huey_consumer jupiter.huey"
Here's what I want to achieve:
1. huey should run as a background process when my Flask app is deployed.
2. supervisor should handle automatic start/stop of the huey process.
I solved this problem by doing the following in an ebextensions file called 002_supervisor.conf. This is for django but I'm sure it could be adapted for flask.
files:
/usr/local/etc/supervisord.conf:
mode: "000755"
owner: root
group: root
content: |
[unix_http_server]
file=/tmp/supervisor.sock ; (the path to the socket file)
[supervisord]
logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false ; (start in foreground if true;default false)
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL for a unix socket
[include]
files = /usr/local/etc/*.conf
[inet_http_server]
port = 127.0.0.1:9001
/etc/init.d/supervisord:
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash
# Source function library
. /etc/rc.d/init.d/functions
# Source system settings
if [ -f /etc/sysconfig/supervisord ]; then
. /etc/sysconfig/supervisord
fi
# Path to the supervisorctl script, server binary,
# and short-form for messages.
supervisorctl=/usr/local/bin/supervisorctl
supervisord=${SUPERVISORD-/usr/local/bin/supervisord}
prog=supervisord
pidfile=${PIDFILE-/tmp/supervisord.pid}
lockfile=${LOCKFILE-/var/lock/subsys/supervisord}
STOP_TIMEOUT=${STOP_TIMEOUT-60}
OPTIONS="${OPTIONS--c /usr/local/etc/supervisord.conf}"
RETVAL=0
start() {
echo -n $"Starting $prog: "
daemon --pidfile=${pidfile} $supervisord $OPTIONS
RETVAL=$?
echo
if [ $RETVAL -eq 0 ]; then
touch ${lockfile}
$supervisorctl $OPTIONS status
fi
return $RETVAL
}
stop() {
echo -n $"Stopping $prog: "
killproc -p ${pidfile} -d ${STOP_TIMEOUT} $supervisord
RETVAL=$?
echo
[ $RETVAL -eq 0 ] && rm -rf ${lockfile} ${pidfile}
}
reload() {
echo -n $"Reloading $prog: "
LSB=1 killproc -p $pidfile $supervisord -HUP
RETVAL=$?
echo
if [ $RETVAL -eq 7 ]; then
failure $"$prog reload"
else
$supervisorctl $OPTIONS status
fi
}
restart() {
stop
start
}
case "$1" in
start)
start
;;
stop)
stop
;;
status)
status -p ${pidfile} $supervisord
RETVAL=$?
[ $RETVAL -eq 0 ] && $supervisorctl $OPTIONS status
;;
restart)
restart
;;
condrestart|try-restart)
if status -p ${pidfile} $supervisord >&/dev/null; then
stop
start
fi
;;
force-reload|reload)
reload
;;
*)
echo $"Usage: $prog {start|stop|restart|condrestart|try-restart|force-reload|reload}"
RETVAL=2
esac
exit $RETVAL
"/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_huey.sh" :
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
# Get django environment variables
env=`cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g' | sed 's/%/%%/g'`
env=${env%?}
# Create huey configuration script
hueyconf="[program:huey]
; Set full path to celery program if using virtualenv
command=/opt/python/current/app/production.py run_huey
user=nobody
numprocs=1
stdout_logfile=/var/log/huey.log
stderr_logfile=/var/log/huey.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 60
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
environment=$env"
# Create the celery supervisord conf script
echo "$hueyconf" | tee /usr/local/etc/huey.conf
# Update supervisord in cache without restarting all services
/usr/local/bin/supervisorctl reread
/usr/local/bin/supervisorctl update
# Start/Restart huey through supervisord
/usr/local/bin/supervisorctl -c /usr/local/etc/supervisord.conf restart huey
commands:
01_start_supervisor:
command: '/etc/init.d/supervisord restart'
leader_only: true