Slow Sidekiq in development only

Question

Running Sidekiq 6.1.0 on Rails 5.0.7.2 with Sidekiq concurrency set to 12, Redis server 4.0.14 and Redis gem 4.2.1.

There seems to be some delay (seconds) between queueing/executing jobs that we didn't see with Rails 4.2.9. Note, this behavior only occurs in development, production seems to do just fine.

An example worker:

class SidekiqTestWorker
  include Sidekiq::Worker

  sidekiq_options(
    queue: "default",
  )

  def perform
    puts "Hello from Sidekiq!"
  end
end

Running 1000.times { SidekiqTestWorker.perform_async } in a Rails console takes around a second to execute all jobs with Rails 4.2.9, but with Rails 5.0.7.2 it takes several minutes to complete. Worth mentioning that we tried running with the same Sidekiq version (5.2.8) with only Rails diffing between the tries with the same result.

A snippet from the Sidekiq worker logs shows the behavior (note the timestamp):

16:52:00 sidekiq_worker.1 | Hello from Sidekiq!
16:52:00 sidekiq_worker.1 | Hello from Sidekiq!
16:52:00 sidekiq_worker.1 | Hello from Sidekiq!
16:52:00 sidekiq_worker.1 | Hello from Sidekiq!
16:52:00 sidekiq_worker.1 | Hello from Sidekiq!
16:52:00 sidekiq_worker.1 | Hello from Sidekiq!
16:52:00 sidekiq_worker.1 | Hello from Sidekiq!
16:52:00 sidekiq_worker.1 | Hello from Sidekiq!
16:52:00 sidekiq_worker.1 | Hello from Sidekiq!
16:52:00 sidekiq_worker.1 | Hello from Sidekiq!
16:52:00 sidekiq_worker.1 | Hello from Sidekiq!
16:52:01 sidekiq_worker.1 | Hello from Sidekiq!
16:52:04 sidekiq_worker.1 | Hello from Sidekiq!
16:52:04 sidekiq_worker.1 | Hello from Sidekiq!
16:52:04 sidekiq_worker.1 | Hello from Sidekiq!
16:52:04 sidekiq_worker.1 | Hello from Sidekiq!
16:52:04 sidekiq_worker.1 | Hello from Sidekiq!
16:52:04 sidekiq_worker.1 | Hello from Sidekiq!
16:52:04 sidekiq_worker.1 | Hello from Sidekiq!
16:52:04 sidekiq_worker.1 | Hello from Sidekiq!
16:52:04 sidekiq_worker.1 | Hello from Sidekiq!
16:52:04 sidekiq_worker.1 | Hello from Sidekiq!
16:52:04 sidekiq_worker.1 | Hello from Sidekiq!
16:52:05 sidekiq_worker.1 | Hello from Sidekiq!
16:52:05 sidekiq_worker.1 | Hello from Sidekiq!
16:52:05 sidekiq_worker.1 | Hello from Sidekiq!

Any ideas what can be done to fix this?

Answer 1

In the Sidekiq 5.0 release notes:

Sidekiq 5.0 contains a reworked job dispatch and execution core to integrate better with the new Rails 5.0 Executor.

The Rails Executor is single-threaded in development mode so it can hot reload job code changes. Sidekiq can only execute one job at a time.

The only way to fix this is to enable eager loading in config/environment/development.rb but that will disable code reloading too.