Search code examples
amazon-web-servicesamazon-ec2emr

Launch EMR and kill it after running one JOB (automatically)


I want to run a job each day automatically on Spark EMR, but i need to optimize my budget and use EMR only when I need it.

So, is there a way to run an EMR (AWS) just for one job, then kill all instances after the job is finished (each X day).


Solution

  • I just discovered the data pipeline service and I founded that this service allows to schedule and run spark jobs. I think that this is the best solution for this use case. https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-manage-recurring.html