Random ActiveRecord::StatementInvalid errors on Rails 4 app on Heroku

Question

My Rails 4 app works fine locally, and also when it's deployed to Heroku in production mode... until I access the app from a second client... then after a few clicks I start getting the following strange sequence of errors (see below).

The exact number of clicks varies a little but it always happens eventually... and but only after a request from a second client. For example, if I restart the app and only play with it on my phone nothing happens. But as soon as I load a couple pages from my laptop... crash

Errors

NOTE: These screenshots are with RAILS_ENV=development so that the actual errors appear.

I'm not sure if I'm understanding these errors correctly, but it looks like ? is not getting properly substituted in the query.

Environment details

• Rails 4.2
• hosted on Heroku
• Ruby 2.2.x
• MySQL (via ClearDB add-on)

My Gemfile:

source 'https://rubygems.org'


# Bundle edge Rails instead: gem 'rails', github: 'rails/rails'
gem 'rails', '4.2.3'
# Use SCSS for stylesheets
gem 'sass-rails', '~> 5.0'
# Use Uglifier as compressor for JavaScript assets
gem 'uglifier', '>= 1.3.0'
# Use CoffeeScript for .coffee assets and views
gem 'coffee-rails', '~> 4.1.0'
# See https://github.com/sstephenson/execjs#readme for more supported runtimes
# gem 'therubyracer', platforms: :ruby

# Use jquery as the JavaScript library
gem 'jquery-rails'
# Turbolinks makes following links in your web application faster. Read more: https://github.com/rails/turbolinks
gem 'turbolinks', '~> 2.3.0'
# Build JSON APIs with ease. Read more: https://github.com/rails/jbuilder
gem 'jbuilder', '~> 2.0'
# bundle exec rake doc:rails generates the API under doc/api.
gem 'sdoc', '~> 0.4.0', group: :doc
gem 'devise'
gem "paperclip"
gem 'aws-sdk'
gem 'dotenv-rails', :groups => [:development, :test]
gem 'foundation-rails'
# Access an IRB console on exception pages or by using <%= console %> in views
gem 'web-console', '~> 2.0', group: :development
gem 'mysql'
gem 'rails_12factor', group: :production
gem 'puma'
group :development, :test do
  # Call 'byebug' anywhere in the code to stop execution and get a debugger console
  gem 'byebug'
  # Use sqlite3 as the database for Active Record
  gem 'sqlite3'    
  # Spring speeds up development by keeping your application running in the background. Read more: https://github.com/rails/spring
  gem 'spring'
end

Here's my Procfile:

web: bundle exec puma -t 5:5 -p ${PORT:-3000} -e ${RACK_ENV:-development}

Here's my database.yml

# SQLite version 3.x
#   gem install sqlite3
#
#   Ensure the SQLite 3 gem is defined in your Gemfile
#   gem 'sqlite3'
#
default: &default
  adapter: sqlite3
  pool: 5
  timeout: 5000

development:
  <<: *default
  database: db/development.sqlite3

# Warning: The database defined as "test" will be erased and
# re-generated from your development database when you run "rake".
# Do not set this db to the same as development or production.
test:
  <<: *default
  database: db/test.sqlite3

production:
  url: <%= ENV['DATABASE_URL'] %>

Here's config/production.rb:

Rails.application.configure do
  # Settings specified here will take precedence over those in config/application.rb.

  # Code is not reloaded between requests.
  config.cache_classes = true

  # Eager load code on boot. This eager loads most of Rails and
  # your application in memory, allowing both threaded web servers
  # and those relying on copy on write to perform better.
  # Rake tasks automatically ignore this option for performance.
  config.eager_load = true

  # Full error reports are disabled and caching is turned on.
  config.consider_all_requests_local       = false
  config.action_controller.perform_caching = true

  # Enable Rack::Cache to put a simple HTTP cache in front of your application
  # Add `rack-cache` to your Gemfile before enabling this.
  # For large-scale production use, consider using a caching reverse proxy like
  # NGINX, varnish or squid.
  # config.action_dispatch.rack_cache = true

  # Disable serving static files from the `/public` folder by default since
  # Apache or NGINX already handles this.
  config.serve_static_files = ENV['RAILS_SERVE_STATIC_FILES'].present?

  # Compress JavaScripts and CSS.
  config.assets.js_compressor = :uglifier
  # config.assets.css_compressor = :sass

  # Do not fallback to assets pipeline if a precompiled asset is missed.
  config.assets.compile = false

  # Asset digests allow you to set far-future HTTP expiration dates on all assets,
  # yet still be able to expire them through the digest params.
  config.assets.digest = true

  # `config.assets.precompile` and `config.assets.version` have moved to config/initializers/assets.rb

  # Specifies the header that your server uses for sending files.
  # config.action_dispatch.x_sendfile_header = 'X-Sendfile' # for Apache
  # config.action_dispatch.x_sendfile_header = 'X-Accel-Redirect' # for NGINX

  # Force all access to the app over SSL, use Strict-Transport-Security, and use secure cookies.
  # config.force_ssl = true

  # Use the lowest log level to ensure availability of diagnostic information
  # when problems arise.
  config.log_level = :debug

  # Prepend all log lines with the following tags.
  # config.log_tags = [ :subdomain, :uuid ]

  # Use a different logger for distributed setups.
  # config.logger = ActiveSupport::TaggedLogging.new(SyslogLogger.new)

  # Use a different cache store in production.
  # config.cache_store = :mem_cache_store

  # Enable serving of images, stylesheets, and JavaScripts from an asset server.
  # config.action_controller.asset_host = 'http://assets.example.com'

  # Ignore bad email addresses and do not raise email delivery errors.
  # Set this to true and configure the email server for immediate delivery to raise delivery errors.
  # config.action_mailer.raise_delivery_errors = false

  # Enable locale fallbacks for I18n (makes lookups for any locale fall back to
  # the I18n.default_locale when a translation cannot be found).
  config.i18n.fallbacks = true

  # Send deprecation notices to registered listeners.
  config.active_support.deprecation = :notify

  # Use default logging formatter so that PID and timestamp are not suppressed.
  config.log_formatter = ::Logger::Formatter.new

  # Do not dump schema after migrations.
  config.active_record.dump_schema_after_migration = false

  config.serve_static_files = true

  config.paperclip_defaults = {
    storage: :s3,
    s3_credentials: {
      bucket: ENV.fetch('S3_BUCKET'),
      access_key_id: ENV.fetch('ACCESS_KEY_ID'),
      secret_access_key: ENV.fetch('SECRET_ACCESS_KEY'),
      s3_region: ENV.fetch('S3_REGION'),
      s3_host_name: 's3-us-west-2.amazonaws.com'
    }
  }


end

Things I tried:

• restarting all dynos (works for a minute but then problem comes back)
• switching to gem mysql2 (couldn't even get this running) See answer below
• switching to gem puma (following this documentation
• connecting to DB using client (everything seems to work fine)
• confirmed that the parameters of the request are correct a la this question (the request params look fine)
• Reading this Heroku / Rails 4 documentation many times

Similar questions that didn't solve my problem

• insert to database error on heroku but worked locally, ActiveRecord::StatementInvalid
• Heroku + Rails Error ActiveRecord::StatementInvalid
• Constant 500 Errors on Heroku

Answer 1

I took a second crack at deploying with the mysql2 gem, and this time it worked. The secret sauce comes from this question

TLDR; You need to specify and older version of mysql2 in order to be compatible with newer versions of Rails. Yea it's weird.

gem 'mysql2', '~> 0.3.18'

After deploying with this gem I can no longer reproduce the problem. I don't understand it, but it worked.

If someone submits an answer that explains what's going on I'll accept it.