AWS Trigger after 100 new files are added to s3

Question

I want a trigger to start a glue job, once after 100 new files(new since the last trigger was fired) are added to s3. How can this be achieved?

Answer 1

You can add events for object creation in S3 buckets:

S3 buckets -> Properties -> Event notification

Now, this does an event for each object created. It cannot do every 100. So, to go around that, you can do the following:

1. Create DynamoDB table which will use on-demand capacity mode. In that table, use just 1 object, that will have fields ID and count.
2. Create a Lambda function that will be called on object creation from the S3 bucket. What you will do there is check the count in DynamoDB. If it's less than 99, increase the count. If it's 99, reset to 0 and trigger the glue job (since this addition would be 100).