So with one dependency I've entered a level of entanglement that I can't escape. I hate to think what will happen when I bring in the commented jars:
libraryDependencies ++= Seq(
// "org.apache.avro" % "avro" % "1.8.1" excludeAll ExclusionRule(organization = "log4j"),
// "org.apache.kafka" %% "kafka" % "0.10.0.0",
"org.apache.hive" % "hive-jdbc" % "1.2.2"
excludeAll ExclusionRule(organization = "log4j")
exclude("org.apache.hadoop", "hadoop-yarn-api"),
"log4j" % "log4j" % "1.2.16"
)
Using sbt assembly, I am getting the following deduplicate problem:
[error] (*:assembly) deduplicate: different file contents found in the following:
[error] C:\Users\G517329\.ivy2\cache\org.datanucleus\datanucleus-api-jdo\jars\datanucleus-api-jdo-3.2.6.jar:plugin.xml
[error] C:\Users\G517329\.ivy2\cache\org.datanucleus\datanucleus-core\jars\datanucleus-core-3.2.10.jar:plugin.xml
[error] C:\Users\G517329\.ivy2\cache\org.datanucleus\datanucleus-rdbms\jars\datanucleus-rdbms-3.2.9.jar:plugin.xml
Where I'm stuck is trying to find a merge strategy that allows these three jars to happily coexist in one fat jar. I've tried several variations of the strategy below, but am making no progress:
assemblyMergeStrategy in assembly := {
case PathList("javax", "transaction", xs @ _*) => MergeStrategy.first
case PathList(xs @ _*) if xs.last endsWith "plugin.xml" => MergeStrategy.discard
// case PathList("org", "datanucleus", "datanucleus-api-jdo", xs @ _*) => MergeStrategy.last
// case PathList("org", "datanucleus", "datanucleus-rdbms", xs @ _*) => MergeStrategy.last
case PathList("META-INF", xs @ _*) => MergeStrategy.discard
case x =>
val oldStrategy = (assemblyMergeStrategy in assembly).value
oldStrategy(x)
}
I must have some misunderstanding of how this works because it seems to me that the second line should find every plugin.xml
file in every jar and nuke it.
Has anyone successfully included hive-jdbc in a fat jar?
UPDATE:
case "plugin.xml" => MergeStrategy.discard //or .last
should work, I would think, but that throws:
[error] (*:assembly) java.util.NoSuchElementException
So I load up the project this morning and give it one more try. This time
case "plugin.xml" => MergeStrategy.last
does not throw, and I have a fat jar.
Guh.
UPDATE:
I don't know where the problem lies (Intellij, SBT Console, sbt-assembly), but there is some caching going on that makes it really difficult to troubleshoot merge issues. I'm finding that the only reliable way to ensure that changes I make to the code in build.sbt are actually applied is to shut down the entire IDE and re-open it.