Search code examples
javaapache-sparksbtignite

Failed to load class name properties in Apache Ignite

I am trying to start an Ignite Context using the Ignite-Spark plugin, for Ignite version 2.2.0. This is how I am declaring the context:

val igniteContext:IgniteContext=new IgniteContext(sparkSession.sparkContext,()=>new IgniteConfiguration().setDiscoverySpi(new TcpDiscoverySpi().
      setLocalAddress("127.0.0.1").setLocalPort(48511).setIpFinder(new TcpDiscoveryMulticastIpFinder().
      setAddresses(new util.ArrayList[String]))).setCommunicationSpi(new TcpCommunicationSpi().setLocalPort(48512)),true)

However I get this error when the context is attempting to start:

IgniteException: Failed to load class names properties file packaged with ignite binaries 
[file=META-INF/classnames.properties, 
ldr=org.apache.spark.util.MutableURLClassLoader@2bbaf4f0]

I read here http://apache-ignite-developers.2346864.n4.nabble.com/classnames-properties-file-is-out-of-date-td2213.html that "'classnames.properties' file is located in ignite-core/META-INF folder. The file is internally used by Ignite marshallers."

I am not too familiar with Ignite's internals. Could this be from the location of my Ignite jar dependency? I used sbt assembly to build my .jar file.

Thank you for your time. T

Solution

  • Turns out I was causing this issue. In order to prevent merge conflicts when running sbt assembly, I added the following merge strategy to my build.sbt file 

    name := "Spark_Ignite_Project"

version := "0.1"

scalaVersion := "2.11.8"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.2.0"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.2.0"
libraryDependencies += "org.apache.spark" %% "spark-streaming" % "2.2.0"
libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.2.0"
libraryDependencies += "org.apache.ignite" % "ignite-core" % "2.2.0"
libraryDependencies += "org.apache.ignite" % "ignite-spring" % "2.2.0"
libraryDependencies += "org.apache.ignite" % "ignite-indexing" % "2.2.0"
libraryDependencies += "org.apache.ignite" % "ignite-log4j" % "2.2.0"


assemblyMergeStrategy in assembly := {
case PathList("META-INF", xs @ _*) => MergeStrategy.discard
case x => MergeStrategy.first
}

    This basically discards all files inside the META-INF folder, which was causing the exception.

    By using the "provided" label on the dependencies that I didn't need to package, everything went well.