Search code examples
apache-sparkapache-spark-sqlspark-launcher

Spark Launcher is not able to trigger spark Job?

I am running a spark standalone cluster in which I have 3 nodes. One is master and other two are workers. I copied the job in all three nodes at location /home/bizruntime/***.jar in all three and when I submit the job using spark-submit in master the job is running.

But my cluster is on remote location and I can not access master node terminal every time. So I think of using Spark Launcher to trigger the job form my laptop which is a Windows machine. But when I run my spark launcher program in Eclipse in my laptop then it starts a driver which I can see in Spark UI but then it gives error. When I check the error in log it because of not finding jar in nodes.

The path which I am giving in appResource() function Spark launcher is /home/bizruntime/***.jar this. But they are searching the jar in $Eclipse_Worksapce/home/bizruntime/***.jar

My question is how to give absolute path in it without $Eclipse_Worksapce.

package com.biz.Remote_Start;

import java.io.IOException;

import org.apache.spark.launcher.SparkAppHandle;
import org.apache.spark.launcher.SparkLauncher;

/**
 * Hello world!
 *
 */
public class App {
  public static void main(String[] args) {

    try {
      Process handle = new SparkLauncher()
          .setAppResource("home/bizruntime/Read_from_ADL-0.0.1-SNAPSHOT.jar")
          .setMainClass("com.biz.Read_from_ADL.App")
          .setMaster("spark://192.168.1.154:7077")
          .setDeployMode("cluster")
          .launch();

      handle.waitFor();
    } catch (Exception e) {

      System.out.println(e.getMessage());
    }

  }
}

enter image description here

Solution

  • Incase jar file exists in local file system, please specify the the location like file:///home/bizruntime/Read_from_ADL-0.0.1-SNAPSHOT.jar