Search code examples
bashamazon-web-servicesemr

How is running a script using aws emr script-runner different from running it from bash?


I have used script-runner on aws emr, and given that it may look very basic (and maybe stuid) question, but I read many documents and noone answers why we need a script runner in emr, when all it does is executing a script in the master node. Can the same script not be run using a bash?


Solution

  • When you are running your script in bash, you need to have the script locally and also you need to set all the configurations to work as you expect it.

    With the script-runner you have more options, for example, run it as part of your cluster launch command, as well execute a script that is hosted remotely in S3. See the example from the EMR documentations: http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-hadoop-script.html