How can I tell unicorn to understand Heroku's signals?

Question

Perhaps you've seen this...

2012-03-07T15:36:25+00:00 heroku[web.1]: Stopping process with SIGTERM
2012-03-07T15:36:36+00:00 heroku[web.1]: Stopping process with SIGKILL
2012-03-07T15:36:36+00:00 heroku[web.1]: Error R12 (Exit timeout) -> Process failed to exit within 10 seconds of SIGTERM
2012-03-07T15:36:38+00:00 heroku[web.1]: Process exited with status 137

This is a well known problem when running unicorn on heroku...

• heroku uses SIGTERM for graceful shutdown
• unicorn uses SIGTERM for quick shutdown

Can I tell heroku to send SIGQUIT? Or can I tell unicorn to treat SIGTERM as graceful shutdown?

Answer 1

This is a hack, but I've successfully created a unicorn config file that traps the TERM signal, preventing unicorn from receiving it and performing its quick shutdown. My signal handler then sends QUIT signal back to itself to trigger the unicorn graceful shutdown.

Tested with Ruby 1.9.2, Unicorn 4.0.1 and 4.2.1, Mac OS X.

listen 9292
worker_processes 1

# This is a hack.  The code is run with 'before_fork' so it runs
# *after* Unicorn installs its own TERM signal handler (which makes
# this highly dependent on the Unicorn implementation details).
#
# We install our own signal handler for TERM and simply re-send a QUIT
# signal to our self.
before_fork do |_server, _worker|
  Signal.trap 'TERM' do
    puts 'intercepting TERM and sending myself QUIT instead'
    Process.kill 'QUIT', Process.pid
  end
end

One concern is that (I believe) this signal handler is inherited by worker processes. But, the worker process installs its own TERM handler, which should overwrite this one, so I would not expect any issue. (See Unicorn::HttpServer#init_worker_process @ lib/unicorn/http_server.rb:551.

Edit: one more detail, this block that installs the signal handler will run once per worker process (because before_fork), but this merely redundant and won't affect anything.