This might seem like a silly question, but I figured someone here on StackOverflow might have some ideas around this. What the hell, right?
I'm using Heroku workers with 1X Dynos to run Resque. Sometimes I get this error: Redis::TimeoutError: Connection timed out. It happens in the redis gem; here's the stacktrace:
Redis::TimeoutError Connection timed out
vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/connection/ruby.rb:55 rescue in _read_from_socket
vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/connection/ruby.rb:48 _read_from_socket
vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/connection/ruby.rb:41 gets
vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/connection/ruby.rb:273 read
vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/client.rb:245 block in read
vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/client.rb:233 io
vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/client.rb:244 read
vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/client.rb:175 block in call_pipelined
vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/client.rb:214 block (2 levels) in process
vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/client.rb:340 ensure_connected
vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/client.rb:204 block in process
vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/client.rb:286 logging
vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/client.rb:203 process
vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/client.rb:174 call_pipelined
vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/client.rb:146 block in call_pipeline
vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/client.rb:273 with_reconnect
vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/client.rb:144 call_pipeline
vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis.rb:2101 block in pipelined
vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis.rb:37 block in synchronize
vendor/ruby-2.1.5/lib/ruby/2.1.0/monitor.rb:211 mon_synchronize
vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis.rb:37 synchronize
vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis.rb:2097 pipelined
vendor/bundle/ruby/2.1.0/gems/redis-namespace-1.5.1/lib/redis/namespace.rb:413 namespaced_block
vendor/bundle/ruby/2.1.0/gems/redis-namespace-1.5.1/lib/redis/namespace.rb:265 pipelined
vendor/bundle/ruby/2.1.0/gems/resque-1.25.2/lib/resque.rb:214 push
vendor/bundle/ruby/2.1.0/gems/resque-1.25.2/lib/resque/job.rb:153 create
vendor/bundle/ruby/2.1.0/gems/resque_solo-0.1.0/lib/resque_ext/job.rb:7 create_solo
vendor/bundle/ruby/2.1.0/gems/resque-1.25.2/lib/resque.rb:317 enqueue_to
vendor/bundle/ruby/2.1.0/gems/resque-1.25.2/lib/resque.rb:298 enqueue
We have resque-retry set up, but I guess it doesn't matter if the enqueue call can't even connect to redis.
My current solution is to wrap every Resque.enqueue call with begin/rescue so that we can re-try the enqueue call (as per https://github.com/resque/resque/issues/840). But isn't there a better way?
Heroku Redis allows you to change your instance timeout and maxmemory-policy settings. Theses settings will be kept across upgrades and HA failover.
From documentation:
The timeout setting sets the number of seconds Redis waits before killing idle connections. A value of zero means that connections will not be closed. The default value is 300 seconds (5 minutes). You can change this value using the CLI:
$ heroku redis:timeout maturing-deeply-2628 --seconds 60Timeout for maturing-deeply-2628 (REDIS_URL) set to 60 seconds. Connections to the redis instance will be stopped after idling for 60 seconds.
Looks like in case of resque the 0 (do not use connection time out) would be the best choise (--seconds 0).