Search code examples
javaapache-sparkjunitspark-submit

Run spark application in Junit

I have a java application which uses Apache Spark. I run it using spark-submit.

The format of the command I use is:

bin/spark-submit \  
--master spark://sparkmaster:7077 \  
--class com.ic.spark.sto.test.Runner \ 
/spark-test/target/spark-test-2.0.jar \   
/spark-test/resources/smallfile.csv \ 
/spark-test/resources/config.properties \ 
true false false

I want to run this as a JUnit. I've created a test class and copied the contents of the main method(from com.ic.spark.sto.test.Runner) method into a test there. When I run it I get an error that suggests the core-site.xml can't be found. At the minute I am trying to run this on a machine with a local spark install. I think that my application doesnt know where spark is this is. How do I change this so it can find spark and execute properly?

Solution

  • I copied and pasted the contents of my main into a JUnit class. I then created a TestRunner clas whick kicks off the JUnit.

    To execute I use the same command as above but pass --class com.ic.spark.sto.test.TestRunner instead of --class com.ic.spark.sto.test.Runner

    The core-site.xml file was in the wrong location.