Stop AWS lambda event source from pushing events if downstream service is down

Question

Is there a way to dynamically stop consuming events, when using aws lambda's built-in event source mapping? In the example diagram I would rely on the Big Service's healthceck to make that decision.

So far I know that if Big Service is down, I could retry processing and eventually put the message in a DLQ. I would prefer to keep the messages in the original queue and thus preserve their order without having to manage processing from DLQ and the FIFO when Big Survice is back.

Answer 1

I found a way to achieve this via Lambda's reserved concurrency.

As stated in the docs:

To throttle a function, set the reserved concurrency to zero. This stops any events from being processed until you remove the limit.

Lambda SDK has a handy method to set the concurrency.

putFunctionConcurrency(params = {}, callback) ⇒ AWS.Request 

And when the downstream service is back, I could delete that setting and resume at the previous pace:

 deleteFunctionConcurrency(params = {}, callback) ⇒ AWS.Request 

My design now is to have a second lambda function monitoring the downstream service's health. When downstream is down, I will set the reserved concurrency to 0, and when it is back up, I will delete the concurrency setting. I am still thinking if I can make the function trigger on a cloudwatch event or trigger it on a time interval, but that is a different question.