Search code examples
ruby-on-railsrubysidekiq

Sidekiq push_bulk to batch

I'm trying to use Sidekiq batches to group a set of related jobs together. However, the batch prematurely fires on complete callback's since the jobs method can't push all the jobs to Redis fast enough. The sidekiq documentation at https://github.com/mperham/sidekiq/wiki/Batches says this can be resolved by using the Sidekiq::Client.push_bulk method but the documentation is unclear on how to push to a batch in bulk. Can someone share an example of how to use push_bulk in the context of a batch?

Solution

  • Assume you want to process 100 users in a batch, one user ID per job.

    b = Sidekiq::Batch.new
b.jobs do
  users = User.select(:id).limit(100).map(&:id) # users is [1, 2, 3, etc...]
  args = users.map {|uid| [uid] } # args is [[1], [2], [3], etc...]
  Sidekiq::Client.push_bulk('class' => YourWorker, 'args' => args)
end