sidekiq runs in bursts on production

Question

In development, it runs as I would expect it, having 5 threads (limited at the moment due to redis connection limit) it averages at about 5-7 processes running, depending if the worker has to do anything or not (sometimes a worker would decide not to work, since the object it is working on was updated less than a few days ago)

on production, it behaves differently. It seems to run in bursts of around 400, and then immediatly reschedules the workers and waits a bit and then shoots a burst again

The workers work with facebook api (koala gem), which for this I use sidekiq-throttler (https://github.com/gevans/sidekiq-throttler)

with the options

sidekiq_options throttle: { threshold: 50, period: 60.seconds , key: ->(user_id){ "facebook:#{user_id}"}  }

I am using heroku and redislabs (free plan at the moment) with the procfile

web: bundle exec puma -C config/puma.rb
worker: bundle exec sidekiq -c 5

and sidekiq setup:

Sidekiq.configure_server do |config|
  config.redis = { :url => "#{ENV['REDISCLOUD_URL']}", :namespace => 'sidekiq'}
  config.server_middleware do |chain|
    chain.add Sidekiq::Throttler, storage: :redis
  end
end

Sidekiq.configure_client do |config|
  config.redis = { :url => "#{ENV['REDISCLOUD_URL']}", :namespace => 'sidekiq' }
end

is this a known symptom for something?

Answer 1

Looks like it's being throttled, as expected.