Search code examples
ruby-on-railsherokuconnection-poolingsidekiq

Why are we out of database connections on Heroku?

We have a Rails app on Heroku with Sidekiq and are running out of database connections.

ActiveRecord::ConnectionTimeoutError: could not obtain a database  
connection within 5.000 seconds (waited 5.000 seconds)

Heroku stuff:

Database plan: Standard0 (120 connections)

Web dynos: 2 Standard-2X

Worker dynos: 1 Standard-2X

heroku config:

MAX_THREADS: 5
(DB_POOL not set)
(WEB_CONCURRENCY not set)

Procfile:

web: bundle exec puma -C config/puma.rb
worker: bundle exec sidekiq

database.yml:

...

production:
  url:  <%= ENV["DATABASE_URL"] %>
  pool: <%= ENV["DB_POOL"] || ENV['MAX_THREADS'] || 5 %>

puma.rb:

# https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server#adding-puma-to-your-application

workers Integer(ENV['WEB_CONCURRENCY'] || 2)
threads_count = Integer(ENV['MAX_THREADS'] || 2)
threads threads_count, threads_count

preload_app!

rackup      DefaultRackup
port        ENV['PORT']     || 3000
environment ENV['RACK_ENV'] || 'development'

on_worker_boot do
  # Worker specific setup for Rails 4.1+
  # See: https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server#on-worker-boot
  ActiveRecord::Base.establish_connection
end

sidekiq.yml:

---
:concurrency: 25
:queues:
  - [default]

We also have a couple of rake tasks that fire every 10 minutes, and they finish within a second or two.

The problem seems to happen when we do a lot of message processing in sidekiq. We do something like:

  1. get article headlines from a 3rd party web service
  2. insert each headline into the db inside a single transaction
  3. create a message in sidekiq for each headline (worker.perform_async)
  4. each message is processed, hits an endpoint to get the body and updates the body (can take .5 - 3 seconds)

While number 4 is happening we see the connection issue.

My understanding is we are way, way, way below the connection limit with our configuration above, but did we do something incorrectly? Is something just consuming the pool? Any help would be great, thanks.

Sources:

Solution

  • You are sharing 5 DB connections among 25 Sidekiq threads. Set DB_POOL to 25 or Sidekiq's concurrency to 5.