Search code examples
rabbitmqamqp

Getting result of a long running task with RabbitMQ

I have a scenario where a client sends an http request to download a file. The file needs to be dynamically generated and typically takes 5-15 seconds. Therefore I am looking into a solution that splits this operation in 3 http requests.

  1. First request triggers the generation of the file.
  2. The client polls the server every 5 seconds to check if file is ready to download
  3. When the response to the poll request is positive, the client starts downloading the file

To implement this I am looking into Message Queue solutions like RabbitMQ. They seem to provide a reliable framework to run long running tasks asynchronously. However after reading the tutorials on RabbitMQ, I am not sure how will I receive the result of the operation.

Here is what I've in mind:

A front end server receives requests from clients and it posts messages to RabbitMQ as required. This front end server will have 3 endpoints

/generate
/poll
/download

When client invokes /generate with a GET parameter say request_uid=AAA, the front end server will post a message to RabbitMQ with the request_uid in the payload. Any free worker will subsequently receive this message and start generating the file corresponding to AAA.

Client will keep polling /poll with request_uid=AAA to check if task was complete.

When task is complete client will call /download with request_uid=AAA expecting to download the file.

The question is how will the /poll and /download handlers of the front end server will come to know about the status of the file generation job? How can RabbitMQ communicate the result of the task back to the producer. Or do I have to implement such mechanism outside RabbitMQ? (Consumer putting its results in a file /var/completed/AAA)

Solution

  • You hit the nail on the head with your last line:

    (Consumer putting its results in a file /var/completed/AAA)

    Your server has to coordinate multiple jobs and the results of their work. Therefore you will need some form of "master repository" which contains an authoritative record of what has been finished already. Copying completed files into a special directory is a reasonable and simple way of doing exactly that.

    It doesn't necessarily need RabbitMQ or any messaging solution, either. Your server can farm out jobs to those workers any way it wishes: by spawning processes, using a thread pool, or indeed by producing AMQP events which end up in a broker and get sucked down by "worker" queue consumers. It's up to your application and what is most appropriate for it.