Search code examples
apache-sparksbtsbt-assembly

spark compilation fails with sbt-assembly 0.13.8

I'm trying to build a spark app after upgrading sbt-assembly to 0.13.8. This is the content of the build.sbt that i have right now. But this doesn't work. It seems to be a problem with the default mergeStrategy. I'am getting around 600+ deduplicate errors 

name := "volumeApp"

version := "0.0.1"

scalaVersion := "2.10.4" 

libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-streaming-kafka" % "1.3.1",
    "org.apache.spark" %% "spark-core" % "1.3.1",
    "org.apache.spark" %% "spark-streaming" % "1.3.1",
    "org.apache.kafka" %% "kafka" % "0.8.2.1",
    "com.datastax.spark" %% "spark-cassandra-connector" % "1.3.0-M1",
    "com.github.vonnagy" %% "service-container-metrics-reporting" % "1.0.1" exclude("com.codahale.metrics", "metrics-core"),
    "joda-time" % "joda-time" % "2.7",
    "log4j" % "log4j" % "1.2.14"
)

assemblyJarName in assembly := "inventoryVolume.jar"

assemblyMergeStrategy in assembly := {
    case PathList("META-INF", "jboss-beans.xml") => MergeStrategy.first
    case PathList("META-INF", "mailcap") => MergeStrategy.discard
    case PathList("META-INF", "maven", "org.slf4j", "slf4j-api", xa @ _*) => MergeStrategy.rename
    case PathList("META-INF", "ECLIPSEF.RSA") => MergeStrategy.discard
    case PathList("META-INF", "mimetypes.default") => MergeStrategy.first
    case PathList("com", "datastax", "driver", "core", "Driver.properties") => MergeStrategy.last
    case PathList("com", "esotericsoftware", "minlog", xx @ _*) => MergeStrategy.first
    case PathList("plugin.properties") => MergeStrategy.discard
    case PathList("javax", "activation", xw @ _*) => MergeStrategy.first
    case PathList("org", "apache", "hadoop", "yarn", xv @ _*) => MergeStrategy.first
    case PathList("org", "apache", "commons", xz @ _*) => MergeStrategy.first
    case PathList("org", "jboss", "netty", ya @ _*) => MergeStrategy.first
    case x =>
        val baseStrategy = (assemblyMergeStrategy in assembly).value
        baseStrategy(x)
}

I have attached the errors over here http://pastebin.com/T6HRJ6Kv Can someone help me out .. I am completely new to build tools . I tried reading the sbt guide but was lost in no time.

Solution

  • It seems that you are trying to build an assembly jar which includes multiple versions of the akka actor system. There are a few possible solutions, one would be to exclude it from your dependencies (see http://www.scala-sbt.org/0.13/docs/Library-Management.html for an idea) or alter your merge strategy but I'd recommend the excluding dependency approach.