Search code examples
web-servicesmessage-queuebackground-processironworkerironmq

When to use a message queue and when to use a cloud background-worker

When would I use a message queue like ironMQ and when would i use a job processing worker like ironWorker?

I have just started researching into these two topics and I am finding it hard to distinguish between the two uses. I understand a worker is more or less a sandbox that will run a program in a different environment outside of the app server to increase the user experience. I also understand that a message queue is much like its database alternative whereby a task is added to a queue and then another server/programming listens for that task and will then process it. However, although I think I understand what they are I am having trouble distinguishing when I would use each one and why.

If I understand correctly I would use a worker for a task such as image processing. But then why can't I use a message queue for that and more importantly why not? Surely I could just have an image URL queued in ironMQ and then have another programming retrieve and process it. In my mind that seems like an extra step so I would avoid that.

A message queue seems fairly pointless to me for common tasks when a worker is available. Surely for non-intensive tasks like posting a comment I could have a worker do that?

I may have misconceived the difference between each tool and if so please set me straight. Otherwise, please help.

Solution

  • They are very closely related so I can understand the confusion. They are both queue based systems, one being a message queue, one being a task/job queue. Here's a general rule of thumb:

    • You'd use IronMQ if you want to run the consumer/worker processes/servers.
    • You'd use IronWorker if you want Iron.io to take care of the consumer/worker processes/servers. You also get a bunch of other features/benefits using IronWorker, not just the fact that you don't have to manage your own servers and scale them, etc.

    So no, you don't need a message queue if you're using IronWorker because IronWorker is your message queue + your processing of that queue.

    Not to add any confusion, but some people use IronWorker and IronMQ together too, with workers pulling messages off IronMQ. This pattern is good for very short tasks to amortize the setup/teardown of a worker (making database connections or whatever the worker has to do to setup).