For a new project we are looking at using Microservices together with RabbitMQ or Kafka. For both technologies I have the same question, the answer might however differ.
Consider three events:
We want to use these events to "duplicate" data among services. When using one consumer, all messages will be executed in the correct order and the database will be consistent.
However, when this one consumer gets to slow at processing the messages, one might want to add another consumer in parallel. At this point it is uncertain that event 1 is executed before event 3, which may lead to an inconsistent database (delete first, create after).
Found some information here about the subject but both solution seem hard to implement. How would it be possible to scale up these consumers? Is there any difference in how one would handle this using RabbitMQ or Kafka?
For Kafka-based implementation, you just need to make sure that all events belonging to same "logical group" (id?) are landing at the same partition - in this case you can add more consumers (up to number of partitions) to handle events in parallel, but every consumer will handle events only for their logical group(s).
You can achieve this behavior by selecting correct key/partitioning approach - either use ID as a key, or come with more sophisticated partitioning implementation.