When spark is running test with sbt. I get this exception :
18:58:49.049 [sparkDriver-akka.actor.default-dispatcher-2] ERROR akka.actor.ActorSystemImpl - Uncaught fatal error from thread [sparkDriver-akka.remote.default-remote-dispatcher-5] shutting down ActorSystem [sparkDriver]
java.lang.VerifyError: (class: org/jboss/netty/channel/socket/nio/NioWorkerPool, method: createWorker signature: (Ljava/util/concurrent/Executor;)Lorg/jboss/netty/channel/socket/nio/AbstractNioWorker;) Wrong return type in function
at akka.remote.transport.netty.NettyTransport.<init>(NettyTransport.scala:283) ~[akka-remote_2.11-2.3.4-spark.jar:na]
at akka.remote.transport.netty.NettyTransport.<init>(NettyTransport.scala:240) ~[akka-remote_2.11-2.3.4-spark.jar:na]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_45]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_45]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_45]
at java.lang.reflect.Constructor.newInstance(Constructor.java:422) ~[na:1.8.0_45]
at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$2.apply(DynamicAccess.scala:78) ~[akka-actor_2.11-2.3.4-spark.jar:na]
at scala.util.Try$.apply(Try.scala:192) ~[scala-library-2.11.7.jar:0.13.8]
at akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:73) ~[akka-actor_2.11-2.3.4-spark.jar:na]
at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84) ~[akka-actor_2.11-2.3.4-spark.jar:na]
at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84) ~[akka-actor_2.11-2.3.4-spark.jar:na]
at scala.util.Success.flatMap(Try.scala:231) ~[scala-library-2.11.7.jar:0.13.8]
at akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:84) ~[akka-actor_2.11-2.3.4-spark.jar:na]
at akka.remote.EndpointManager$$anonfun$9.apply(Remoting.scala:692) ~[akka-remote_2.11-2.3.4-spark.jar:na]
at akka.remote.EndpointManager$$anonfun$9.apply(Remoting.scala:684) ~[akka-remote_2.11-2.3.4-spark.jar:na]
at scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:728) ~[scala-library-2.11.7.jar:0.13.8]
at scala.collection.Iterator$class.foreach(Iterator.scala:742) ~[scala-library-2.11.7.jar:0.13.8]
at scala.collection.AbstractIterator.foreach(Iterator.scala:1194) ~[scala-library-2.11.7.jar:0.13.8]
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) ~[scala-library-2.11.7.jar:0.13.8]
at scala.collection.AbstractIterable.foreach(Iterable.scala:54) ~[scala-library-2.11.7.jar:0.13.8]
at scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:727) ~[scala-library-2.11.7.jar:0.13.8]
at akka.remote.EndpointManager.akka$remote$EndpointManager$$listens(Remoting.scala:684) ~[akka-remote_2.11-2.3.4-spark.jar:na]
at akka.remote.EndpointManager$$anonfun$receive$2.applyOrElse(Remoting.scala:492) ~[akka-remote_2.11-2.3.4-spark.jar:na]
at akka.actor.Actor$class.aroundReceive(Actor.scala:465) ~[akka-actor_2.11-2.3.4-spark.jar:na]
at akka.remote.EndpointManager.aroundReceive(Remoting.scala:395) ~[akka-remote_2.11-2.3.4-spark.jar:na]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) [akka-actor_2.11-2.3.4-spark.jar:na]
at akka.actor.ActorCell.invoke(ActorCell.scala:487) [akka-actor_2.11-2.3.4-spark.jar:na]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) [akka-actor_2.11-2.3.4-spark.jar:na]
at akka.dispatch.Mailbox.run(Mailbox.scala:220) [akka-actor_2.11-2.3.4-spark.jar:na]
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) [akka-actor_2.11-2.3.4-spark.jar:na]
...
First thing I had think that it is a wrong version of netty imported by one of my sub lib.
After looking to my dependency-graph I found 4 differents version of netty.
3.6.6
3.8.0
3.9.3
4.0.23
Theses version are mostly imported by spark himself :o
The 4.0.23 is directly imported by spark and the 3.8.0 by its sub-dependency Akka-remote
I tried to exclude netty dependency from all of my sub-dependencies with excludeAll( ExclusionRule(organization = "io.netty")) and add each netty version one by one. But it didn't resolve the problem.
I also tried to exclude all com.typesafe.akka deps to use Akka-remote 2.4.0 I get the same problem.
The netty version needed by Akka-remote is the 3.8.0. And even forcing this one doesn't work. I also forced my project to use the spatial spark Akka version Akka-remote:3.8.0-spark that don't change anything.
for info my build.sbt
libraryDependencies ++= Seq(
"com.datastax.cassandra" % "cassandra-driver-core" % "2.1.7.1" excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
"com.datastax.spark" %% "spark-cassandra-connector" % "1.4.0" excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
"com.datastax.spark" %% "spark-cassandra-connector-embedded" % "1.4.0" % Test excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
"xxx.xxxxx" %% "shed" % "0.10.0-MOK-1848-DEBUG-SNAPSHOT" excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
"com.twitter" %% "util-collection" % "6.27.0" excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
"io.circe" %% "circe-core" % "0.2.0-SNAPSHOT" excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
"io.circe" %% "circe-generic" % "0.2.0-SNAPSHOT" excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
"io.circe" %% "circe-jawn" % "0.2.0-SNAPSHOT" excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
"io.netty" % "netty" % "3.6.6.Final",
//"com.typesafe.akka" % "akka-remote_2.11" % "2.3.4",
"org.apache.cassandra" % "cassandra-all" % "2.1.5" excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
"org.apache.cassandra" % "cassandra-thrift" % "2.0.5" excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
"org.apache.spark" %% "spark-streaming-kafka" % "1.4.0" excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
"org.apache.spark" %% "spark-streaming" % sparkVersion % "provided" excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
"org.apache.spark" %% "spark-core" % sparkVersion % "provided" excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
"org.apache.spark" %% "spark-sql" % sparkVersion % "provided" excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
"org.scalaz.stream" % "scalaz-stream_2.11" % "0.7.3" % Test excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka")),
"org.specs2" %% "specs2-core" % "3.6.1-scalaz-7.0.6" % "test" excludeAll( ExclusionRule(organization = "io.netty"), ExclusionRule(organization = "com.typesafe.akka"))
)
As you see I try to exclude maximum things for my tries
and the subproject shed contain
"com.github.scopt" %% "scopt" % "3.3.0" ,
"com.typesafe.akka" %% "akka-testkit" % "2.3.8" % "test",
"com.typesafe.play" %% "play-test" % "2.3.8" % "test",
"com.tinkerpop.gremlin" % "gremlin-java" % gremlinVersion,
"com.tinkerpop" % "pipes" % gremlinVersion,
"com.thinkaurelius.titan" % "titan-core" % titanVersion,
"com.thinkaurelius.titan" % "titan-cassandra" % titanVersion,
"com.thinkaurelius.titan" % "titan-berkeleyje" % titanVersion,
"com.netaporter" %% "scala-uri" % "0.4.8",
"com.github.nscala-time" %% "nscala-time" % "1.8.0",
"com.mandubian" %% "play-json-zipper" % "1.2",
"com.michaelpollmeier" %% "gremlin-scala" % "2.6.1",
"com.ansvia.graph" %% "blueprints-scala" % "0.1.61-20150416-SNAPSHOT",
"io.kanaka" %% "play-monadic-actions" % "1.0.1" exclude("com.typesafe.play", "play_2.11"),
"org.scalaz" %% "scalaz-concurrent" % "7.0.6",
"com.chuusai" %% "shapeless" % "2.3.0-SNAPSHOT",
("io.github.jto" %% "validation-json" % "1.0").exclude("org.tpolecat", "tut-core_2.11"),
"org.parboiled" %% "parboiled" % "2.1.0",
"com.typesafe.scala-logging" %% "scala-logging" % "3.1.0",
"ch.qos.logback" % "logback-classic" % "1.1.2",
"xxx.xxxxxxxxxx" %% "chuck" % "0.9.0-SNAPSHOT",
"xxx.xxxxxxxxxx" %% "shed-graph" % "0.9.0-MOK-1848-SNAPSHOT" exclude("com.thinkaurelius.titan", "titan-core"),
"io.circe" %% "circe-core" % "0.2.0-SNAPSHOT",
"io.circe" %% "circe-generic" % "0.2.0-SNAPSHOT",
"io.circe" %% "circe-jawn" % "0.2.0-SNAPSHOT"
It's a trap !
netty organisation changed in the past. from org.jboss.netty to io.netty, but they contain the same package.
exclude("org.jboss.netty", "netty") resolve my problem.