How can I make resque worker process other jobs while current job is sleeping?

Question

Each task I have work in a short bursts, then sleep for about an hour and then work again and so on until the job is done. Some jobs may take about 10 hours to complete and there is nothing I can do about it.

What bothers me is that while job is sleeping resque worker would be busy, so if I have 4 workers and 5 jobs the last job would have to wait 10 hours until it can be processed, which is grossly unoptimal since it can work while any other worker is sleeping. Is there any way to make resque worker to process other job while current job is sleeping?

Currently I have a worker similar to this:

class ImportSongs
  def self.perform(api_token, songs)
    api = API.new api_token

    songs.each_with_index do |song, i|
      # make current worker proceed with another job while it's sleeping
      sleep 60*60  if i != 0 && i % 100 == 0

      api.import_song song
    end
  end
end

Answer 1

It looks like the problem you're trying to solve is API rate limiting with batch processing of the import process.

You should have one job that runs as soon as it's enqueued to enumerate all the songs to be imported. You can then break those down into groups of 100 (or whatever size you have to limit it to) and schedule a deferred job using resque-scheduler in one hour intervals.

However, if you have a hard API rate limit and you execute several of these distributed imports concurrently, you may not be able to control how much API traffic is going at once. If you have that strict of a rate limit, you may want to build a specialized process as a single point of control to enforce the rate limiting with it's own work queue.