Goal of zero downtime, how to use upstart with sockets & (g)unicorn:

Question

My goal is zero downtime deployments for an ecommerce app, and I'm trying to this in the best way possible.

I'm doing this on a nginx/unicorn/django setup as well as a nginx/unicorn/rails setup for a separate server.

My strategy is to set preload_app=true in my guincorn.py/unicorn.rb file, then reload by sending a USR2 signal to the PID running the server. This forks the process and it's children and a pre_fork/before_fork can pick up on this and send a subsequent QUIT signal.

Here's an example of what my pre_fork is doing in the guincorn version:

# ...

pidfile='/opt/run/my-website/my-website.pid'

# socket doesn't come back after QUIT
bind='unix:/opt/run/my-website/my-website.socket'

# works, but I'd prefer the socket for security
# bind='localhost:8333'

# ...

def pre_fork(server, worker):
    old_pid_file = '/opt/run/my-website/my-website.pid.oldbin'

    if os.path.isfile(old_pid_file):
        with open(old_pid_file, 'r') as pid_contents:
            try:
                old_pid = int(pid_contents.read())
                if old_pid != server.pid:
                    os.kill(old_pid, signal.SIGQUIT)
            except Exception as err:
                pass

pre_fork=pre_fork

And here's a selection from my sysv script which performs the reload:

DESC="my website"
SITE_PATH="/opt/python/my-website"
ENV_PATH="/opt/env/my-website"
RUN_AS="myuser"

SETTINGS="my.settings"
STDOUT_LOG="/var/log/my-website/my-website-access.log"
STDERR_LOG="/var/log/my-website/my-website-error.log"
GUNICORN="/opt/env/my-website/bin/gunicorn.py"

CMD="$ENV_PATH/bin/python $SITE_PATH/manage.py run_gunicorn -c $GUNICORN >> $STDOUT_LOG 2>>$STDERR_LOG"

sig () {
  test -s "$PID" && kill -$1 `cat $PID`
}

run () {
  if [ "$(id -un)" = "$RUN_AS" ]; then
    eval $1
  else
    su -c "$1" - $RUN_AS
  fi
}

reload () {
  echo "Reloading $DESC"
  sig USR2 && echo reloaded OK && exit 0
  echo >&2 "Couldn't reload, starting '$DESC' instead"
  run "$CMD"
}
action="$1"
case $action in
reload)
  reload
  ;;
esac

I chose preload_app=true, for the zero-downtime appeal. Since the workers have the app preloaded into memory, then as long as I switch processes correctly, it should simulate a zero downtime result. That's the thinking anyway.

This works where I'm listening to through a port but I haven't been able to get it work over a socket.

My questions are the following:

• Is this how the rest of you are doing this?
• Is there a better way, for example with HUP somehow? My understanding is you can't use preload_app=true with HUP though.
• Is it possible to do this using a socket? My socket keeps going away on the QUIT and never coming back. My thinking is that a socket is more secure because you have to have access to the filesystem.
• Is anyone doing this with upstart rather than sysv? I'd ideally like to do that and I saw an interesting way of accomplishing that by flocking the PID. It's a challenge with upstart because once the exec-fork from gunicorn/unicorn takes over, upstart is no longer monitoring the process it was originally managing and needs to be re-established somehow.

Answer 1

You should look at unicornherder from my colleagues at GDS, which is specifically designed to manage this:

Unicorn Herder is a utility designed to assist in the use of Upstart and similar supervisors with Unicorn.