I have an application that has 3 instances/replicas (three applications of the same kind are run simultaneously). They all consume the same queue, so RabbitMQ delivers messages in round robin fashion. All three instances share a single database that they r/w to.
This works great when the queue receives, let's say, new-user message, delivers it to one instance, and that instance saves the user to the common database.
The problem:
Given a message new-user, I want one instance to save it to the database (as before), but now I want all three to invalidate their caches. So I actually need all instances to receive the same event, but only one of them to act differently.
How can you design a system that allows these requirements?
The best way I can think of is to have two exchanges (fanout and direct), new-user message to be sent to a separate application which duplicates it and sends it to fanout and direct exchanges. All instances listen to both exchanges, and depending on the message handler, decides on what to do (if the message is from fanout exchange: invalidate the cache, if it's from direct exchange, well, only one would receive it anyways).
There can be a problem with your approach. If messages from fanout exchange will arrive before user was actually created, your instances may invalidate cache too early.
I think you should do the following:
direct exchange and worker queue where all your user creators are getting messages in round robin, like you have it now.user created to a topic or fanout exchange, where all user creators are listening. Then all of the instances (including sender) will receive the user created event and will invalidate their cache.