Search code examples
eclipseapache-sparkhadoop-yarn

Could not parse Master URL

i will be thankful if anyone answer my question. part of my code is:

public static void main(String[] args) {
    // TODO Auto-generated method stub
    SparkConf conf= new SparkConf().setAppName("AirPollution").setMaster("yarn");
    SparkContext context=new SparkContext(conf);

how can i fix this error?

Exception in thread "main" org.apache.spark.SparkException: Could not parse Master URL: 'yarn'
Solution

  • I find the soultion. any body that has this problem should add spark-yarn.jar to the build path. the important issue is that IT SHOULD BE THE LAST VERSION, IF YOU USE SPARK 2+. for example spark-yarn2.11 worked for me