Search code examples
amazon-web-servicesamazon-sqs

AWS SQS groups in FIFO queues


So from the SQS documentation, consumer can be scaled horizontally, if group -ids are added to messages. I assume that these group-ids are dynamic.

If a consumer is initially handling messages of group-id A and B, and a big inflow of A comes in, would messages of group B be automatically routed to another consumer if no messages of type B are currently in-flight ? How is the consumer client routing determined, is it dynamic, or a fixed hash ? eg: if new group-ids are added is it ensured that the load is distributed evenly across consumers ?

Is there a way to get group-wise counts of the queue size ?


Solution

  • There is no "consumer client routing" in Amazon SQS.

    Rather, consumers are responsible for asking the SQS queue for messages.

    It is your responsibility to scale and balance the clients appropriately.

    From Avoid having a large backlog of messages with the same message group ID - Amazon Simple Queue Service:

    For FIFO queues, there can be a maximum of 20,000 inflight messages (received from a queue by a consumer, but not yet deleted from the queue). If you reach this quota, Amazon SQS returns no error messages. If your queue has a large backlog of 20,000 or more messages with the same message group ID, FIFO queues might be unable to return the messages that have a different message group ID but were sent to the queue at a later time until you successfully consume the messages from the backlog.

    This is only a problem if the messages are currently 'being processed'.