Search code examples
shellspark-shell

Unable to run spark sql through shell script

I am unable to query a table in spark through shell script. But if i am running through command line, i am able to get the result. Problem arises when i insert those command in shell and trying to run.

Created a shell script :

vi test.sh

Inserted below spark shell command

spark-shell

val results =sqlContext.sql("SELECT * from table_name ")

results.show()

It is entering into spark shell but not running the below two command

val results =sqlContext.sql("SELECT * from table_name ")

results.show()

Solution

  • You can use Except to get spark-shell working in bash script.

    OR create a file with .scala and copy all your spark commands there.

    val results =sqlContext.sql("SELECT * from table_name ")

results.show()

System.exit(0)

    use ' spark-shell -i script_name.scala ' to run your script in bash or directly on linux terminal.

    System.exit(0)----- to get out from spark-shell