How to find out if a set of Resque jobs have finished?

Question

I have the following code:

[23, 45, 69, 20].each do |page_id|
  Resque.enqueue(ProcessPage, page_id)
end

I would like to know when all those jobs have finished processing to notify the user via email. My first attempt was to notify the user at the very end of the job, but that would work just individually for each job.

I have found this gem resque-status, which basically let's you do this:

job_ids = []
[23, 45, 69, 20].each do |page_id|
  job_ids << ProcessPage.create(page_id)
end

And then you can check the status of a job by doing:

status = Resque::Plugins::Status::Hash.get(job_id)

What confuses me is... When should I do that checking for all the statuses? I mean, would I be doing a while checking the status of all the jobs? That would make the server to timeout, any ideas how I can do this?

Answer 1

I'm not going to paste any code here because it looks like you're pretty familiar with how jobs should be written but think about the following idea:

You should have a resque job which gets and array of ids to process. This resque jobs creates a single resque job for each of the ids so they will be processed in parallel. After creating the resque jobs for specific ids it will create another resque jobs which gets the list of ids and periodically checks on them to see if they're finished. When they're done, this job will send the user an email.

With this paradigm you enjoy all of the worlds:

1. Each ID has its own job.
2. Parallel execution.
3. Status check is done on the background thus the user won't get timeout.

UPDATE:

A pseudo code for the check resque job:

class CheckerJob
  @queue = :long

  def self.perform(ids)
    finished = []
    while finished.size < ids.size
      ids.each do |id|
        finished << id if job_finished?(id)
      end
      sleep 10
    end
    send_email_to_user
  end
end

Now all you left to do is to implement both job_finished?(id) and send_email_to_user methods.