How to prioritize queues using Resque and Redis

Question

My application has multiple queues and I'm trying to prioritize specific queues.

resque-pool.yml

development:
  "high_priority, med_priority, low_priority": 1

According to the resque-pool documentation (https://github.com/nevans/resque-pool) example:

resque-pool.yml

foo: 1
bar: 2
"foo,bar,baz": 1

production:
  "foo,bar,baz": 4

"This will start up seven worker processes, one exclusively for the foo queue, two exclusively for the bar queue, and four workers looking at all queues in priority. With the config above, this is similar to if you ran the following:"

rake resque:work RAILS_ENV=production QUEUES=foo &
rake resque:work RAILS_ENV=production QUEUES=bar &
rake resque:work RAILS_ENV=production QUEUES=bar &
rake resque:work RAILS_ENV=production QUEUES=foo,bar,baz &
rake resque:work RAILS_ENV=production QUEUES=foo,bar,baz &
rake resque:work RAILS_ENV=production QUEUES=foo,bar,baz &
rake resque:work RAILS_ENV=production QUEUES=foo,bar,baz &

I think queues are prioritized by the order they are in within the yaml files but would want to verify it as well.

How would I manually verify that the jobs within the highest_priority queue are being prioritized over jobs in the med_priority and low_priority queue? Am I missing something in order to prioritize them correctly?

Answer 1

You have it right. As the main Resque documentation states:

Resque doesn't support numeric priorities but instead uses the order of queues you give it. We call this list of queues the "queue list."

Let's say we add a warm_cache queue in addition to our file_serve queue. We'd now start a worker like so:

$ QUEUES=file_serve,warm_cache rake resque:work

When the worker looks for new jobs, it will first check file_serve. If it finds a job, it'll process it then check file_serve again. It will keep checking file_serve until no more jobs are available. At that point, it will check warm_cache. If it finds a job it'll process it then check file_serve (repeating the whole process).

In this way you can prioritize certain queues.

Note that, in a resque-pool setup, each process that spawns will follow its own priority (so in the example config it's possible bar jobs will get processed before foo jobs if the process that only processes bar is free while the others are busy). That's a feature, though, not a bug!

As for how you can test it, simply enqueue a bunch of jobs in each queue and watch as they get processed in resque-web (or the logs).