We have a Workflow (Step Function) that is triggered via an EventBridge rule and inside that Workflow is a lambda that creates a record in a third party vendor system. Assuming the third party vendor API does not have a way to enforce uniqueness of a record, we are wondering if it's possible to achieve idempotency (no duplicate record creations in vendor) without introducing synchronous processing e.g. a FIFO queue/limit lambda concurrency.
As EventBridge does not gaurantee only once delivery but instead guarantees at least once delivery, we encountered an issue with duplicate records being created in the vendor system due to duplicate events being processed at the same time (5ms~ apart).
I am wondering if it's possible to prevent this without introducing a FIFO queue or limiting lambda concurrency to 1, if not we will proceeed with that and add a check in the lambda to see if the record already exists in vendor system or if the record is processed/processing on our side.
My understanding is a control check on the uniqueness of the record will not suffice while concurrency exists (e.g. 2 lambdas concurrently processing the same record checking the vendor API/Dynamo/SQL will return the same result).
I am pretty sure we have our answer/solution but just checking if I have not overlooked something to do this.
Lambda Powertools provides an idempotency module to help with this type of scenario.
It is currently available in Python and Java programming languages.
It handles some of the difficult edge cases and is backed by DynamoDB.