Is it possible to have multiple consumers for an SQS FIFO queue, when my use case restricts me to process messages in creation order and only once?

Question

i have a service which will consume from SQS queue, and multiples instances of this service will be running in parallel on EC2 or Kubernetes pods to avoid single point of failure, now my queue is a FIFO queue, i wanted to preserve the ordering if msg's and process exactly once. Now each instance of my app will have a SQS consumer. Lets say there are 2 msgs in the queue m1 and m2 and 2 consumers c1 and c2, if c1 receives the m1, and c2 receives m2 and c2 processes m2 before m1 could be processed by c1, this will create a problem for me, is there any way to avoid this while having multiple consumers

i need to know if there is a way to preserve the order when having multiple consumers or maybe replace SQS with some other service to achieve my goal

Answer 1

When a message is sent to an Amazon SQS FIFO queue, the sender supplies a MessageGroupId that identifies the 'group' of messages.

For example, imagine a fleet of buses driving around a city. Each bus sends its location to the queue every few minutes. The MessageGroupId contains the bus identifier. It is important to consume the messages from each bus in order otherwise a map might show the bus going backwards on a map.

To achieve this, the FIFO queue will do the following:

• Consumer-1 requests some messages from the queue. SQS will supply messages from any bus (that is, with any MessageGroupId). It might provide multiple messages from the same Group.
• These messages are made 'invisible' while the Consumer processes the messages. This is done in case the Consumer fails, in which case the messages will later reappear in the queue.
• Let's say Consumer-2 requests some messages while Consumer-1 is still processing its messages. SQS will NOT provide any messages to Consumer-2 that belong to any of the Message Group IDs that Consumer-1 is currently processing. This ensures that messages are processed in the correct order. For example, if Consumer-1 is processing messages for Bus 1 & Bus 2, then SQS will not provide Consumer-2 with any messages from Bus 1 or Bus 2.

This means that Consumer-2 might process some 'later' messages (eg from Bus 3) that were received after messages from Bus 1. This is okay, since all messages from a given bus will always be processed in order.

So, think of the FIFO queue as actually having lots of 'mini-queues', each with a different MessageGroupId. While one Group's messages are being processed, Consumers can continue processing messages from a different Group.

If your particular situation requires only a single Group of messages that must ALL be processed strictly in order, then subsequent consumers will not be provided any messages until the existing Consumer has finished processing the messages from that single Group. In this scenario, it is not possible to processes messages in parallel.