AMQP routing messages to multiple queues in order

Question

Little Background

• My app publishes messages to an exchange, this app knows nothing about where the message needs to go, or where it's going to go.
• The message needs to travel through a couple steps (queues) in a pipeline. For simplicity we'll call them pre-process process and post-process.

My question is

Is there anything in AMQP (Rabbit-mq specifically) that can help me with the handling of these messages in that order? (pre-process, then process, then post-process).

Solutions I currently know of

1. Handling the routing logic in the services themselves, so the pre-process service would have to know that the next step is process. The services would handle publishing the messages to the next exchange or queue.

   The only issue I have with this, is I don't necessarily want the pre-process service to know, or care about where the messages have to go next. If I need to add in another service in-between pre-process and process, I would have to change app code or configuration in pre-process, and then also make sure that the new service also knows that the next step is process.

2. Using some type of service bus.

   I don't know much about a Service Bus, but I think this is what it's made to handle.
   The only issue I have with a service bus is that all the implementations I have looked at (NServiceBus, MassTransit) look pretty heavyweight. They have their own new sets of terminology, they have lots of features, and if something goes wrong we now need to be experts in this specific bus technology, they seem to add a ton of un-needed complexity to the process.

3. Creating my own router service.

   Each message would contain info in it's headers about which queue it's hit. The router would then be in charge of sending the message to the correct service. Each service would always publish it's messages back to the router when they are finished doing their job.

   Is there any smells with doing something like this? The only issue I see with it, is that it seems like we're basically taking a lot of control away from our queue system which has pretty great routing capabilities.

Any thoughts on the matter, or some examples from the trenches would be great.

Answer 1

Rabbit has great routing capabilities but it does not do service orchestration. It is more the scope of a service bus which can rely on a messaging queue.

You can refine 1. :

1. A message is self-sufficient and contains all the routing logic. For example message can contain an header with all the routing logic associated to a processsing chain. For example a property routings:

   "routings": ["pre-process" , "process", "post-process"]

   So a process step does not need to know the next process step. It pops the first entry of the routings array and sent the next message to this queue. Pretty suitable if the processing step is linear, does not require conditionnal step or historization.

   So each service must contain the routing logic.

The third solution is simpler to manage (separation of concern between service). One service is responsible of the routing and it calls through RabbitMQ the appropriate process step. The smell may be that it needs more messages than the first solution. The cost of this drawback depends on your requirement. In fact to improve this, you will tend towards a service bus which will be a mix of solution 1) and 3).

I used at work the third solution. The processing steps are defined by a state machine.