InvocationTargetException. Cannot cast class X to class X. When invoked in Scala Imain through spark-submit

Question

So, I have the following use case.

I'm simplifying the usage of Spark dataframes for a particular domain by providing a DSL like interface. All this code goes in a fat jar created by maven shade plugin. (fat jar = without spark and hadoop dependencies)

This fat jar has a main class, lets call it JavaMain.

Inside JavaMain, I make a rest call to get a string whose contents are valid DSL.

I instantiate a IMain object with initial Settings object. And I bind a few variables. using the imain.bind method.

However this bind fails with the following error:

Set failed in bind(results, com.dhruv.dsl.DslDataFrame.DSLResults, com.dhruv.dsl.DslDataFrame$DSLResults@7650a5f3)
java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:734)
    at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.callEither(IMain.scala:738)
    at scala.tools.nsc.interpreter.IMain.bind(IMain.scala:625)
    at scala.tools.nsc.interpreter.IMain.bind(IMain.scala:661)
    at scala.tools.nsc.interpreter.IMain.bind(IMain.scala:662)
    at com.thoughtworks.dsl.DSL.run(DSL.scala:44)
    at com.thoughtworks.dsl.JavaMain.run(JavaMain.java:30)
    at com.thoughtworks.dsl.JavaMain.main(JavaMain.java:43)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:665)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassCastException: com.thoughtworks.dsl.DslDataFrame$DSLResults cannot be cast to com.thoughtworks.dsl.DslDataFrame$DSLResults
    at $line3.$eval$.set(<console>:6)
    at $line3.$eval.set(<console>)
    ... 21 more

More context:

I had issues with classpath when trying this out. Although it seems like I haven't been able to resolve them all.

Earlier when creating Setting object, I was doing something like this:

val settings = {
  val x = new Settings()
  x.classpath.value += File.pathSeparator + System.getProperty("java.class.path")
  x.usejavacp.value = true
  x.verbose.value = true
  x
}

However this didn't seem to work as when doing a spark submit this only had the spark and hadoop related jars on the classpath.

I then added the following to the classpath:

val urLs: Array[URL] = Thread.currentThread.getContextClassLoader.asInstanceOf[URLClassLoader].getURLs

and did following:

val settings = {
  val x = new Settings()
  x.classpath.value += File.pathSeparator + urLs(0)
  x.usejavacp.value = true
  x.verbose.value = true
  x
}

This is the code I'm using to bind the objects:

interpreter.bind("notagin", new SomeDummyObject)

This throws the exception I attached earlier. Interestingly the following code works: (i.e. imports and new of the same object inside Interpreter doesn't cause a problem)

interpreter.interpret(
  """
      import com.dhruv.dsl.operations._
      import com.dhruv.dsl.implicits._
      import com.dhruv.dsl.DslDataFrame._
      import org.apache.spark.sql.Column
      import com.dhruv.dsl._
      implicit def RichColumn(column: Column): RichColumn = new RichColumn(column)
      val justdont = new SomeDummyObject()
      justdont.justdontcallme(thatJson)
   """
)

Another detail that I'm aware of and is bothering me is that IMain internally does change the classloader. Not sure if that is causing the issue.

Any help is more than appreciated.

Answer 1

Okay. So we figured out how to solve the problem for us.

I think IMain uses a different classloader to load classes then the one they are supposed to be loaded with. Anyways, the following solves the problem, leaving it for others to have a look.

val interpreter = new IMain(settings){
  override protected def parentClassLoader: ClassLoader = this.getClass.getClassLoader
}