Search code examples
ruby-on-railssidekiq

Sidekiq ActiveJob track spawned jobs in

A lot of time I will have background worker code that looks like this.

class MyJob < ActiveJob::Base

 def perform
   User.find_in_batches.each do |batch|
     MyOtherJobToProcessBatchOfUsers.perform_later(batch)
   end
 end
end

Basically, one jobs that spawns many child jobs. Is there an easy way to know when all of the jobs are finished? I know I could increment a counter in the parent job and have each child job decrement that counter. But as anyone who has tried to do that will tell you, you can run into some complications.

Any help would be appreciated.

Solution

  • This is what Sidekiq Pro's Batch feature does. You are right: it's incredibly difficult to build correctly. I don't know of any open source library that implements it.

    https://github.com/mperham/sidekiq/wiki/Batches http://sidekiq.org/pro/