I am using a combination of SQS and Lambda to perform a bulk activity so that I can do this in a controlled fashion, otherwise I was facing API rate limit exception (API is limited at 150 calls / 5 second).
Once this batch job is finished, I want to be notified of it's completion so that I can send a message or any kind of other notification to a user. How can I achieve this?
Implementation - SQS receives bulk messages, say 10K. A lambda(reserve concurrency set at 18) is configured with SQS as trigger point, batch size of only 1 message. This lambda function performs some computation and logic to update my program's state. (Each function hits around 5 APIs in a single run) This takes around 1 and a half hour to complete entire job and I do not get any API rate limit exception.
There are several ways of doing this.
For example, when your SQS gets initially populate you enable CW Alarm on a CW metric for SQS. The alarm could trigger on ApproximateNumberOfMessagesVisible is zero for a period of time. The alarm triggers SNS which in turn triggers a lambda function. The lambda function would be responsible for verifying that that all messages are correctly processed (e.g. no messages in an associated DLQ queue, results are saved somewhere), disable the alarm and send out some notification.
The other way would be for your processing lambda to check the state of the queue. When the queue has zero messages, it could send SNS to trigger other lambda. The other lambda would again verify if all messages get processed and send out notifications. This could be a bit problematic if lambda functions get invoked in parallel.
Yet another way would to setup a CW Events rule which triggers a lambda every 5 minutes for example. The rule would be enabled when the SQS gets initially populated. The lambda would check the state of the SQS queue, and if its empty and the messages were correctly processed, it would send out notifications, and disable the rule.
You can have obviously combinations of these possibilities. Hope this will be helpful on a pathway to your goal.