Search code examples
scalaintellij-ideaapache-zeppelin

Scala code running on Intellij, but not on Apache Zeppelin notebook

I have developed some scala code to iterate over a java.util.ArrayList[Source], where Source is a case class with two Strings.

If I run my code on IntelliJ, it runs perfectly, when I get it to my Zeppelin Notebook it gives me an error.

My code is:

source_list.forEach(s => if(s.getPersonalDataFlg.equals("YES")){println(s.getPersonalDataID)})

The error in Apache Zeppelin is:

<console>:45: error: missing parameter type

Can someone tell me why is this happening?

Solution

  • Since older versions of Java did not have Lambda's, older versions of Scala had a totally different Function implementation.

    Since you are using forEach on ArrayList which was added in Java 8, it indicates that you are already using a Java version later than Java 8.

    After the introduction of Lambda's in Java 8, the later versions of Scala had their Function implementation based on Java lambda's.

    Now, you source_list is a java.util.ArrayList[Source] which means that source_list.forEach expects a parameter of type Consumer<? super Source> which can be replaced by a Java lambda such as (Source s) -> { }.

    You are providing it with following scala function,

    s => if(s.getPersonalDataFlg.equals("YES")){println(s.getPersonalDataID)

    This gets compiled to the appropriate lambda in case of newer Scala versions but will fail with older scala versions.

    // this will work with Scala 2.12.x but fail with older versions
import java.util.ArrayList

val list = new ArrayList[Int]()

list.add(1)

list.forEach(i => println(i))

    This error indicates that your Zeppelin notebook is running older versions of Scala (most likely 2.11.x or even 2.10.x)

    With older versions of Scala, you have to explicitly create the Consumer and pass that to forEach

    import java.util.ArrayList
import java.util.function.Consumer

val list = new ArrayList[Int]()

list.add(1)


val consumer: Consumer[Int] = new Consumer[Int] {
  override def accept(i: Int): Unit = println(i) 
}

list.forEach(consumer)