I want to run a job each day automatically on Spark EMR, but i need to optimize my budget and use EMR only when I need it.
So, is there a way to run an EMR (AWS) just for one job, then kill all instances after the job is finished (each X day).
I just discovered the data pipeline service and I founded that this service allows to schedule and run spark jobs. I think that this is the best solution for this use case. https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-manage-recurring.html