Launch EC2 instance on S3 upload + process file

Question

I want to automatically launch an EC2 instance whenever a user uploads data to my datadump-input S3 bucket. The EC2 instance should then:

1. Start
2. Download the file from datadump-input and process it with pre-loaded Python scripts
3. Stop

What's the easiest way to do this? Specifically, how do I get the file from the S3 bucket to the EC2 instance? The filename will be different on each upload.

I was thinking Lambda but am open to other services.

Answer 1

I would have S3 send the new object notification to an SQS queue. Configure an auto-scaling group to launch an EC2 instance when the queue depth is greater than 0. On the EC2 instance, have your application poll the SQS queue for messages until there are no more.