How to handle HTTP request to a Message Broker Producer/Consumer?

Question

Let's say you have a POST request with some product as the payload. Traditionally, your HttpRequest lifecycle should end with an HttpResponse carrying the requested action's result, in our case a response saying "Product created" might be enough.

But with a message broker, things might turn like this:

1. The request handler create the appropriate message, CreateProduct(...), and produces it to a topic in the message broker.
2. Then what ???
3. A consumer retrieves and process the message by actually creating the product in a persistent database.
4. Then What ???

What should happens at step 2 ?

If we send a response saying "Your product should be created very soon, keep waiting, we keep you posted":

• How can the client be notified after a response has been sent already ?
• Are we forced to use WebSocket so we can keep the link opened ?

What should happens at step 4 ?

I have my opinion but I would like to know how you handle it in production.

The app that actually created the product can produce a message saying "Product created" to a status topic in the message broker, so the original message's producer can consume it and then notify the client some how. The only way I see it possible is through a WebSocket connection.

So I would like to know if WebSocket is the only way to do Http Request/Response involving a message broker ? and is it reasonable to use a message broker for Http Request/Response ?

Answer 1

You could think of this in a fully asynchronous fashion ( no websocket needed then).

You do an POST Http request and this will create an unique ID associated with your job. This ID will be stored in a database as well, with a status like 'processing'. Besides the ID will be returned to your client.

Your job ID ( and its payload parameters) travels inside Kafka and finally goes to a consumer. This consumer will process the job and commit stuff to external DB ( or whatever). When the job is done you update the job status to 'done' or something like this.

In the meantime, client side, you poll an endpoint that will ask your Job DB state if the job is over or not.

This is a very common way to cover your needs.

Yannick