So I have a situation with a microservice architecture where I need to guarantee that incoming messages that have common identifier would be processed in order they come from kafka:
message2, message1 kafka
------------------------------
|message1 |message2
| |
Instace1 Instance2
In the example below, I have two instances of a service that are processing messages from kafka, but I want to ensure that message2 is only processed after message1.
Apparently, this situation is easily solved by configuring one instance to consume only from a particular partition which would store messages with the common indetifier:
message2, message1 kafka
--------------------------------
| message2
| message1
Instance1 Instance2
Now the order is guaranteed, and message2 will never be processed before message1.
However, I was wondering if this issue could be solved another way, directly in code instead of relying on infrastructure? This looks like it could be a standard problem in microservice architecture but I'm not sure what would be the preferred approach to solve it ?
I'd suggest infrastructure as the more "correct" way to go, but solving this with code should be possible:
If you have a single producer of messages, attach to the message the identifier of the directly-preceding message and before consuming the message make sure you consumed the directly-preceding one before.
If you have multiple producers, this gets a bit more tricky, as you'd have to synchronise the identifiers.
Again, I suggest the infrastructure to be the more "correct" way of solving this (the less code you write, and the less complex, the less bugs you'll have).