Search code examples
ruby-on-railsherokumemory-leaksunicorn

Two instances running on Heroku but only 1 worker. Why?

I have been troubleshooting a memory usage of about 550 Mb on a Heroku Rails-app running on Unicorn which is causing some 2k ms response times.

I looked at my New Relic-graphs and realized I am running two instances but I only have 1 worker and I am only running 1 dyno (Hobby). I don't understand why there are two instances! It seems like I am accidentally using a "ghost" instance. This only happens when I use Unicorn, not on Puma.

Edit: I added a worker to see what happened if I ran 2 workers. This caused 3 instances to be running, according to New Relic, so it does not duplicate it just add one ghost instance.

Once every 10 minutes I run a short scheduled task, which can be seen in the graphs. ENV["WEB_CONCURRENCY"] is not set, by the way.

enter image description here

# Unicorn.rb:
worker_processes Integer(ENV["WEB_CONCURRENCY"] || 1)
timeout 15
preload_app true

before_fork do |server, worker|

  Signal.trap 'TERM' do
    puts 'Unicorn master intercepting TERM and sending myself QUIT instead'
    Process.kill 'QUIT', Process.pid
  end

  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.connection.disconnect!
end  

after_fork do |server, worker|

  Signal.trap 'TERM' do
    puts 'Unicorn worker intercepting TERM and doing nothing. Wait for master to send QUIT'
  end

  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.establish_connection
end

# Proc
web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb

Running heroku ps gives:

heroku ps
=== web (Hobby): bundle exec unicorn -p $PORT -c ./config/unicorn.rb (1)
web.1: up 2018/01/12 11:34:08 +0100 (~ 5h ago)

Is this behavior to be expected or am I doing something terribly wrong here? What could cause the second instance to run? Is it possible to accidentally start two versions on the app on boot?

Solution

  • I removed the scheduler, some gems and the dalli cache and this removed the extra/ghost instance. Then I put them back one after another but it stayed at one instance. I.e. exactly the same setup that before had two instances, now were down to 1 (which makes the most sense).

    The memory consumtion remains the same so I will mark this down as a New Relic bug. Unfortunately.