How to deduplicate events when using RabbitMQ Publish/Subscribe Microservice Event Bus

Question

I have been reading This Book on page 58 to understand how to do asynchronous event integration between microservices.

Using RabbitMQ and publish/subscribe patterns facilitates pushing events out to subscribers. However, given microservice architectures and docker usage I expect to have more than once instance of a microservice 'type' running. From what I understand all instances will subscribe to the event and therefore would all receive it.

The book doesn't clearly explain how to ensure only one of the instances handle the request.

I have looked into the duplication section, but that describes a pattern that explains how to deduplicate within a service instance but not necessarily against them...

Each microservice instance would subscribe using something similar to:

public void Subscribe<T, TH>()
        where T : IntegrationEvent
        where TH : IIntegrationEventHandler<T>
    {
        var eventName = _subsManager.GetEventKey<T>();

        var containsKey = _subsManager.HasSubscriptionsForEvent(eventName);
        if (!containsKey)
        {
            if (!_persistentConnection.IsConnected)
            {
                _persistentConnection.TryConnect();
            }

            using (var channel = _persistentConnection.CreateModel())
            {
                channel.QueueBind(queue: _queueName,
                                    exchange: BROKER_NAME,
                                    routingKey: eventName);
            }
        }

        _subsManager.AddSubscription<T, TH>();
    }

I need to understand how a multiple microservice instances of the same 'type' of microservice can deduplicate without loosing the message if the service goes down while processing.

Answer 1

From what I understand all instances will subscribe to the event and therefore would all receive it.

Only one instance of subscriber will process the message/event. When you have multiple instances of a service running and subscribed to same subscription the first one to pick the message will set the message invisible from the subscription (called visibility timeout). If the service instance is able to process the message in given time it will tell the queue to delete the message and if it's not able to process the message in time , the message will re-appear in queue for any instance to pick it up again.

All standard service bus (rabbitMQ, SQS, Azure Serivce bus etc) provide this feature out of box.

By the way i have read this book and used the above code from eShotContainers and it works the way i described.

You should look into following pattern as well Competing Consumers pattern

Hope that helps!