Search code examples
amazon-web-servicesamazon-s3amazon-emr

AWS EMR process immediately as log lands in S3


My actual implementation includes Device Farm & EMR. Device Farm produces logs and saves them in S3 and I want EMR to immediately pick them up and process (ultimate goal is to put produced structured info to DynamoDB).

What's the best approach? Is it possible to do that without integration of yet another thing which checks if there are no new logs in S3?


Solution

  • You can use events on your S3 bucket. Create an event viz. whenever new object( log file) is created; invoke lambda or SNS notification ( which in turn invokes EMR )