I have a Rails app and a set of NodeJS servers. I use those NodeJS servers to parse URLs using PhantomJS. From my Rails app I know which NodeJS servers are ready to receive a URL to be parsed, and once it is parsed, the result is sent asynchronously back to the Rails server.
My problem is that if I will need a queue in Rails so that the URLs that need to be parsed are queued until some of the NodeJS servers are ready to parse. So, my question is:
What would be a good priority-queue solution in Rails/Ruby that would let me keep checking the queue and the status of the NodeJS servers and send the job from that queue to the free NodeJS server.
I have used Sidekiq before, but I believe that in this case this is not necessary as the Worker would not do any expensive job here, but just delegating from the Rails queue to one of the NodeJS servers.
You could use Redis with BLPOP, for example. Just create a list for each level of priority. Have rails push URLs into the respective list for its priority, and the blocking pop waits for jobs to show up. Simple and lightweight, and redis should be well supported by both platforms in use.
But really, you could adapt almost any centralized storage into a home-rolled priority queue mechanism - for example you could get away with polling a table in a run of the mill RDBMS, as long as it has a priority and status (ready, running, complete, error, etc.) column that you can filter and sort. Not that I'd recommend that solution for a high volume use case, but it is possible.