No applicable constructor/method found for zero actual parameters - Apache Spark Java

Question

I have been facing strange error with my spark job at runtime. I don't see anything wrong with MyBean class, any idea what could be wrong with below driver code? Thanks

Maven Dependencies-

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.10</artifactId>
    <version>2.1.0</version>
</dependency>

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.10</artifactId>
    <version>2.1.0</version>
</dependency>

Driver-

SparkSession spark = SparkSession.builder().config(conf).getOrCreate();
spark.createDataset(Arrays.asList(new MyBean(10),new MyBean(20)),
      Encoders.bean(MyBean.class)).show();

.....

class MyBean implements Serializable {
    int i;
    public MyBean(){}
    public MyBean(int i){this.i=i;}
    public int getI() {return i;}
    public void setI(int i) {this.i = i;}
}

Runtime Exception-

ERROR org.codehaus.commons.compiler.CompileException:org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 43, Column 21: No applicable constructor/method found for zero actual parameters; candidates are: "public int com.ts.spark.datasets.MyBean.getI()" at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:11004) at org.codehaus.janino.UnitCompiler.findMostSpecificIInvocable(UnitCompiler.java:8307) at org.codehaus.janino.UnitCompiler.findIMethod(UnitCompiler.java:8169) at org.codehaus.janino.UnitCompiler.findIMethod(UnitCompiler.java:8071)

Answer 1

Spark needs public JavaBean class. It looks like you are defining the MyBean class in the same class where you are creating SparkSession. There are two options to resolve this issue. First option is - Create a separate class file for MyBean.java public class.

public class MyBean implements Serializable {
    int i;
    //Getters and Setters
}

Second option is - Define MyBean as public static inner class of your main class like below.

public class MyClass {
    public static void main(String[] args) {
        SparkSession spark = ...;
    }

    public static class MyBean implements Serializable {
        int i;  
        //Getters and Setters
    }
}