I am using dynamodb streams to capture changes in the table. I have set the trigger to a lambda function. Insert and delete are working as expected but while updating specifically the hash key or sort key of any document it creates 2 events REMOVE and INSERT. That leads lambda function to process two events instead of one. How can I configure it to send only one event as MODIFY. Or how can I identify REMOVE and INSERT belongs to the same operation? Also, one more problem with this is as I have set batch size and batch window. If the batch size is full, it is possible events come in different batches which will lead to inconsistency.
I tried to merge REMOVE and INSERT event by identifying consecutive sequence numbers. But sometimes they have consecutive sequence numbers and sometimes not.
I'm not sure how you are updating keys in DynamoDB, as that is a forbidden operation, I assume you are using some sort of wrapper or third party library.
You cannot use UpdateItem to update any primary key attributes. Instead, you will need to delete the item, and then use PutItem to create a new item with new attributes.
As you cannot update keys, your library is performing 2 operations which is a Delete (REMOVE) and a Put (INSERT/MODIFY), and this would be as expected.
While you cannot merge the events, you could use Lambda Event Filters to only listen to INSERT/MODIFY events.