Ensuring Sequential Message Processing in Node.js Application

Question

I have a Node.js application that acts as a connector, facilitating message exchange between WhatsApp and another application, let's call it CCM. Here's how it works:

• The connector receives a message from WhatsApp.
• It forwards this message to CCM.
• CCM publishes the message to an SNS topic.
• Upon successfully publishing the message, CCM sends a 200 OK response to the connector, indicating successful asynchronous communication.
• CCM also subscribes to the same SNS topic to receive the published message. Like one endpoint of CCM will publish message to SNS and another endpoint will get that message and process it.
• Upon receiving the message, CCM initiates a chat using Amazon Connect Chat APIs.

Flow Diagram 1 Flow Diagram 2

However, I'm encountering an issue: when multiple messages are sent in quick succession (2-3 messages together), CCM starts multiple chats for the same customer. This happens because CCM takes some time to process the first message, and meanwhile, the subsequent messages arrive quickly. Processing messages contains finding customer from db against the information contains in that message etc.

Questions:

• Is this behavior normal for the pub-sub system?
• Can anyone suggest solution of this problem?
• Shouldn't CCM wait for the acknowledgment of the first message before processing subsequent ones?

Answer 1

It seems your issue is that the CCM processes multiple messages in parallel that are associated with the same customer.

• Is this behavior normal for the pub-sub system?

Yes. SNS sends messages to its subscribers and does not wait for confirmation that the message has been processed.
So, parallel message processing by the CCM can occur.

• Can anyone suggest solution of this problem?
• Shouldn't CCM wait for the acknowledgment of the first message before processing subsequent ones?

To address this, you can implement the following solution:

1. Introduce SQS FIFO
2. Configure your SNS topic as FIFO.
3. Make SQS the subscriber to the SNS topic.
4. Set MessageGroupId as customer ID when sending messages to SNS.
5. Make CCM to pull messages from SQS FIFO.

As a result, the CCM will not process a message for a particular customer if another or same CCM instance is already processing a message with the same MessageGroupId (i.e., customer ID). This guarantees that messages for the same customer are processed sequentially.