What is the best practice to architect tasks processing using AWS?

Question

I am wondering about how to configure AWS Lambda, SNS, and SQS for processing background tasks.

There are three ways I thought.

Option 1. A function called consumer can execute workers by receiving tasks from the queue.

Option 2. Send all tasks to SNS. One worker and one SQS receive and work from SNS.

Option 3. Directly forward the task to one SQS and one lambda from the APP.

The biggest concern is whether to directly invoke Lambda in the app or use task consumer using SQS or SNS.

My idea is from Triggering multiple lambda functions from one SQS trigger

Answer 1

It depends on your current and future requirements:

Options 1: Choosing consumer lambda will allow you to add validations and manipulation in the event. But your consumer lambda will be running until your worker lambdas are running.

Option 2: SNS gives you flexibility to add new events in future and new subscribers as well and your App will have to deal with only SNS.

Option 3: If you are sure in future there will be no such other lambdas. In this case your app need to have the configuration which type of event will go to which SQS.

You can choose any option based on your requirement but I will suggest you to choose option 2 as your app will be required to push notification to SNS only(Single integration). In SNS you can add filters for different types of event. From SNS you can directly trigger lambda as well.

If you do not need output of lambda functions in your App you should use SNS/SQS for async processing.