We have SQS --> aws lambda -->step function setup. For every message in queue lambda is getting triggered which eventually start execution of step function.
Currently for every message, lambda is triggering step function and we wanted to control this throttling. Is there a way to process only one message at a time and once processing is completed then only lambda should poll new message and invoke step function.
We kept batch item = 1 and batch window to 30 sec. Reserved concurrency of lambda aws set to 1. With this configuration also, for parallel step function execution was observed for more than one messages.
Your pattern throttles the Lambda to one concurrent execution, but this doesn't limit your Step Function to one concurrent execution. Why not? Because the StartExecution API is asynchronous. As @MarkB says, your Lambda starts the Step Function execution and exits without waiting for it to finish.
So how do you throttle a Step Function? Here are some options:
You seem to be using event source mapping to integrate SQS with Lambda. The Lambda service manages polling the queue behind the scenes.
Because you aren't in charge of when your Lambda is invoked, the key is checking for in-progress executions with the ListExecutions API, filtering on the RUNNING status. If the length of the response's executions array is 0, you are OK to start a new execution.
Add this logic to your Lambda, calling StartExecution only if there are no running executions.
What to do if an execution is running depends on your use case. The easiest approach is to return the messge to the Queue by throwing an error. But that pattern isn't ideal (multiplied message traffic and failed executions) if your SM is long-running or you often have multiple queued messages. In that case, you could consider adding an arbitrary wait mechanism to your Lambda. Or, manually return the message to the queue with an delay (a SendMessage call with DelaySeconds).
Alternatively, you can take charge of Lambda invocations. Run your Lambda on a schedule, manually pulling a single message from the queue every X seconds with the ReceiveMessage API. The value of X would depend on your use case. X might simply be a number greater than your Step Function execution time. If you'd prefer to minimize processing latency, you could instead choose a more frequent schedule (lower X value), then have your Lambda check for running executions as above.
Finally, you could have a schedule-triggered Lambda pull *multiple* messages from the queue. You'd then wrap your Step Function in a Map State with MaxConcurrency set to 1. The input messages would be processed one at a time.