Search code examples
ruby-on-railsherokuresqueresque-scheduler

Is it safe to use redis for long-time-pending tasks in Rails via resque?

how safe it is to use Resque for long-time-pending tasks in Rails (for example: Resque.enqueue_in(7.days, JobClass)) with resque and resque-scheduler gems? Good to mention that instance is running on Heroku. What is the chance of losing queue?

Solution

  • Semi Persistence

    As per my comments, the downside of using Redis (which Resque uses) is the potentiality of downtime.

    As the "RAM" of web-apps, Redis stores semi persistent data. This basically means it only retains your data whilst it is operating. Any downtime would lose your queue.

    This is the crux of your question -

    can you afford to lose your queue?

    The answer is only one which you can answer.

    There are, however, a number of factors which I would consider:

    1. *Redis isn't a full database... it's a JSON key:value pair store. Key values

      In short, this should only be used to give you a simple mechanism to use snippets of data at particular times. We use Redis to store ID's of users on one of our systems, as an example.

      This means that if you're storing more data inside Redis than simple ID's or other base data, are you using it correctly?

    2. How Important Is Your Queue?

      As with any app, you'll want to retain as much user data as possible.

      The difference, however, is how critical the queue is to your system.

      enter image description here

      I gave the example of a Hotel Reservation system. Perhaps you have "welcome" emails you need to send, "directions" emails to help guests find the hotel, etc. Are these "mission critical"?

      I have designed an email marketing system before. The queuing system for that was extensive; but it wasn't the "queue" that was important - it was all the associative data which went with it.

      Thus, we saved our data in a datatable and only used Redis to store a pure queue of what was going to be sent, and when. We then ran some scripts every minute (I've forgotten the specifics), which went through the Redis queue.

    Bottom line is that I would highly recommend storing the appropriate data in a table, and then only processing the requests as you need them.

    The underpin of the queue is that you need to be able to rebuild it if it goes down. If you're storing your values in redis alone, this will prove to be a major bottleneck.

    Hitting the DB is well worth the data you'll get from it in the long run.