Resque queue size

Question

I am using Resque and Redis to setup a queue for jobs. I am uploading some big files that i'd like to break down into smaller files. Is there a way in Resque to setup queue sizes where it will process smaller chunks of data and break it down into smaller queues?

Answer 1

The way I have done this is to set up a job that does a large query, preferably using find_in_batches. Then, within that block, spawn another Resque job on another queue. Then you can set up multiple workers to listen to the second queue which actually does the work, while Resque still handles "batching" those jobs out.