Search code examples
sbtsbt-assembly

Need a pathlist pattern to make sbt happy


So with one dependency I've entered a level of entanglement that I can't escape. I hate to think what will happen when I bring in the commented jars:

libraryDependencies ++= Seq(
//  "org.apache.avro" % "avro" % "1.8.1" excludeAll ExclusionRule(organization = "log4j"),
//  "org.apache.kafka" %% "kafka" % "0.10.0.0",
  "org.apache.hive" % "hive-jdbc" % "1.2.2"
    excludeAll ExclusionRule(organization = "log4j")
    exclude("org.apache.hadoop", "hadoop-yarn-api"),
  "log4j" % "log4j" % "1.2.16"
)

Using sbt assembly, I am getting the following deduplicate problem:

[error] (*:assembly) deduplicate: different file contents found in the following:
[error] C:\Users\G517329\.ivy2\cache\org.datanucleus\datanucleus-api-jdo\jars\datanucleus-api-jdo-3.2.6.jar:plugin.xml
[error] C:\Users\G517329\.ivy2\cache\org.datanucleus\datanucleus-core\jars\datanucleus-core-3.2.10.jar:plugin.xml
[error] C:\Users\G517329\.ivy2\cache\org.datanucleus\datanucleus-rdbms\jars\datanucleus-rdbms-3.2.9.jar:plugin.xml

Where I'm stuck is trying to find a merge strategy that allows these three jars to happily coexist in one fat jar. I've tried several variations of the strategy below, but am making no progress:

assemblyMergeStrategy in assembly := {
  case PathList("javax", "transaction", xs @ _*) => MergeStrategy.first
  case PathList(xs @ _*) if xs.last endsWith "plugin.xml" => MergeStrategy.discard
  // case PathList("org", "datanucleus", "datanucleus-api-jdo", xs @ _*) => MergeStrategy.last
  // case PathList("org", "datanucleus", "datanucleus-rdbms", xs @ _*) => MergeStrategy.last
  case PathList("META-INF", xs @ _*) => MergeStrategy.discard
  case x =>
    val oldStrategy = (assemblyMergeStrategy in assembly).value
    oldStrategy(x)
}

I must have some misunderstanding of how this works because it seems to me that the second line should find every plugin.xml file in every jar and nuke it.

Has anyone successfully included hive-jdbc in a fat jar?

UPDATE:

case "plugin.xml" => MergeStrategy.discard //or .last should work, I would think, but that throws:

[error] (*:assembly) java.util.NoSuchElementException


Solution

  • So I load up the project this morning and give it one more try. This time

    case "plugin.xml" => MergeStrategy.last

    does not throw, and I have a fat jar.

    Guh.

    UPDATE:

    I don't know where the problem lies (Intellij, SBT Console, sbt-assembly), but there is some caching going on that makes it really difficult to troubleshoot merge issues. I'm finding that the only reliable way to ensure that changes I make to the code in build.sbt are actually applied is to shut down the entire IDE and re-open it.