Updated to include Redis/Resque versions and stack trace (below):
redis (3.0.4)
redis-namespace (1.3.0)
redis (~> 3.0.0)
redis-store (1.1.2)
redis (>= 2.2.0)
resque (1.24.1)
mono_logger (~> 1.0)
multi_json (~> 1.0)
redis-namespace (~> 1.2)
sinatra (>= 0.9.2)
vegas (~> 0.1.2)
resque-scheduler (2.0.1)
redis (>= 2.0.1)
resque (>= 1.20.0)
rufus-scheduler
I'm seeing intermittent Redis::TimeoutError: Connection timed out on heroku while writing a moderately sized array (~200 Fixnums) to the Redis store using the Rails.cache.fetch command.
I'm also using Resque.
I see here that the Redis::Client can receive a timeout option, but I don't see where to pass initialization options to Redis.
I'm using the standard heroku resque.rb:
rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
rails_env = ENV['RAILS_ENV'] || 'development'
resque_config = YAML.load_file(rails_root + '/config/resque.yml')
ENV['REDIS_URI'] = resque_config[rails_env]
Resque.redis = resque_config[rails_env]
Resque.inline = rails_env == 'test'
require 'resque_scheduler'
require 'resque/scheduler'
require 'resque_scheduler/server'
Resque.schedule = YAML.load_file(rails_root + '/config/resque-schedule.yml')
Resque.before_fork do
defined?(ActiveRecord::Base) and
ActiveRecord::Base.connection.disconnect!
end
Resque.after_fork do
defined?(ActiveRecord::Base) and
ActiveRecord::Base.establish_connection
end
I assume that a Redis client is instantiated here. Is this a different client from the one instantiated in production.rb:
rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
rails_env = ENV['RAILS_ENV'] || 'development'
resque_config = YAML.load_file(rails_root + '/config/resque.yml')
config.cache_store = :redis_store, resque_config[rails_env], { expires_in: 14.days }
The options hash there is for Rails.cache options as far as I know. Is a new client instantiated here? How can I pass options to this one?
Updated to include this experiment in the heroku console implying that they are different client instances:
irb(main):002:0> Rails.cache
=> #<ActiveSupport::Cache::RedisStore:0x00000003860e18 @data=#<Redis client v3.0.4 for redis://spinyfin.redistogo.com:9485/0>, @options={:expires_in=>14 days}>
irb(main):003:0> Resque.redis.redis
=> #<Redis client v3.0.4 for redis://spinyfin.redistogo.com:9485/0>
irb(main):004:0> Rails.cache.instance_variable_get(:@data).object_id == Resque.redis.redis.object_id
=> false
Stack trace:
Redis::TimeoutError: Connection timed out
from /app/vendor/bundle/ruby/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:208:in `rescue in io'
from /app/vendor/bundle/ruby/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:206:in `io'
from /app/vendor/bundle/ruby/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:214:in `read'
from /app/vendor/bundle/ruby/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:84:in `block in call'
from /app/vendor/bundle/ruby/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:187:in `block (2 levels) in process'
from /app/vendor/bundle/ruby/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:295:in `ensure_connected'
from /app/vendor/bundle/ruby/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:177:in `block in process'
from /app/vendor/bundle/ruby/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:256:in `logging'
from /app/vendor/bundle/ruby/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:176:in `process'
from /app/vendor/bundle/ruby/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:84:in `call'
from /app/vendor/bundle/ruby/1.9.1/gems/redis-3.0.4/lib/redis.rb:644:in `block in setex'
from /app/vendor/bundle/ruby/1.9.1/gems/redis-3.0.4/lib/redis.rb:36:in `block in synchronize'
from /app/vendor/ruby-1.9.3/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
from /app/vendor/bundle/ruby/1.9.1/gems/redis-3.0.4/lib/redis.rb:36:in `synchronize'
from /app/vendor/bundle/ruby/1.9.1/gems/redis-3.0.4/lib/redis.rb:643:in `setex'
from /app/vendor/bundle/ruby/1.9.1/gems/redis-store-1.1.2/lib/redis/store/interface.rb:17:in `setex'
... 11 levels...
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/cache.rb:299:in `fetch'
...SNIP...
...my code...
...SNIP...
from /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.13/lib/active_record/relation/delegation.rb:6:in `each'
from /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.13/lib/active_record/relation/delegation.rb:6:in `each'
...SNIP...
...my code...
...SNIP...
from (irb):5
from /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/commands/console.rb:47:in `start'
from /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/commands/console.rb:8:in `start'
from /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/commands.rb:41:in `<top (required)>'
from script/rails:6:in `require'
from script/rails:6:in `<main>'irb(main):006:0> ! Heroku client internal error.
! Search for help at: https://help.heroku.com
! Or report a bug at: https://github.com/heroku/heroku/issues/new
Error: Operation timed out (Errno::ETIMEDOUT)
Backtrace: /usr/local/heroku/ruby/lib/ruby/1.9.1/openssl/buffering.rb:121:in `sysread'
/usr/local/heroku/ruby/lib/ruby/1.9.1/openssl/buffering.rb:121:in `readpartial'
/Users/me/.heroku/client/lib/heroku/client/rendezvous.rb:69:in `block in start'
/Users/me/.heroku/client/lib/heroku/client/rendezvous.rb:53:in `loop'
/Users/me/.heroku/client/lib/heroku/client/rendezvous.rb:53:in `start'
/Users/me/.heroku/client/lib/heroku/command/run.rb:132:in `rendezvous_session'
/Users/me/.heroku/client/lib/heroku/command/run.rb:119:in `run_attached'
/Users/me/.heroku/client/lib/heroku/command/run.rb:24:in `index'
/Users/me/.heroku/client/lib/heroku/command.rb:206:in `run'
/Users/me/.heroku/client/lib/heroku/cli.rb:28:in `start'
/usr/local/heroku/bin/heroku:24:in `<main>'
Command: heroku run rails c
Plugins: heroku-redis-cli
Version: heroku-toolbelt/2.39.4 (x86_64-darwin10.8.0) ruby/1.9.3
This isn't much of an answer, but at the recommendation of a colleague, I switched from RedisToGo to openredis and these problems went away immediately.