Search code examples
apache-sparktimestampemr

Spin down after last spark-submit completion (or failure) + xxx time

I'm using Amazon's EMR cluster, but I'm not using the add steps API. I'd like a way to shutdown the cluster if, for example, it has been idle for 3 days.

If I were to build this, I'd likely just run a history command on the cluster and look for the last spark-submit command. Is there a better way?

Solution

  • the easiest way to do this is to check the list of yarn jobs. you can display the running jobs with:

    yarn application -list

    -list gives you some more options too, which you might want to explore.