When running a java program using mvn exec:java, the program runs fine. When I use maven-assembly-plugin to create a single jar file, running the same class causes an issue caused by loading an older version of a dependency.
The error I got is:
Caused by: java.lang.NoSuchFieldError: INSTANCE
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.<clinit>(SSLConnectionSocketFactory.java:144)
at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.getPreferredSocketFactory(ApacheConnectionManagerFactory.java:87)
at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:65)
at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:58)
at com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory.create(ApacheHttpClientFactory.java:50)
at com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory.create(ApacheHttpClientFactory.java:38)
This error is caused because httpclient version loaded was less than 4.3.
I placed this dependency at the top of pom.xml.
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.2</version>
</dependency>
I use an assembly.xml file that looks like this:
<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3 http://maven.apache.org/xsd/assembly-1.1.3.xsd">
<id>jar-with-dependencies</id>
<formats>
<format>jar</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
<dependencySets>
<dependencySet>
<outputDirectory>/</outputDirectory>
<unpack>true</unpack>
<scope>runtime</scope>
<excludes>
<exclude>com.almworks.sqlite4java:libsqlite4java-linux-i386</exclude>
<exclude>com.almworks.sqlite4java:libsqlite4java-linux-amd64</exclude>
<exclude>com.almworks.sqlite4java:sqlite4java-win32-x64</exclude>
<exclude>com.almworks.sqlite4java:sqlite4java-win32-x86</exclude>
<exclude>com.almworks.sqlite4java:libsqlite4java-osx</exclude>
</excludes>
</dependencySet>
<dependencySet>
<outputDirectory>/</outputDirectory>
<unpack>false</unpack>
<includes>
<include>com.almworks.sqlite4java:libsqlite4java-linux-i386</include>
<include>com.almworks.sqlite4java:libsqlite4java-linux-amd64</include>
<include>com.almworks.sqlite4java:sqlite4java-win32-x64</include>
<include>com.almworks.sqlite4java:sqlite4java-win32-x86</include>
<include>com.almworks.sqlite4java:libsqlite4java-osx</include>
</includes>
</dependencySet>
</dependencySets>
</assembly>
Here is the dependency tree
[INFO]
[INFO] --- maven-dependency-plugin:2.1:tree (default-cli) @ yy-importer ---
[INFO] com.mohataher.xx:yy-importer:jar:1.0-SNAPSHOT
[INFO] +- com.mohataher.xx:base:jar:1.0-SNAPSHOT:compile
[INFO] | +- com.thinkaurelius.titan:titan-core:jar:1.0.0:compile
[INFO] | | +- org.glassfish:javax.json:jar:1.0:compile
[INFO] | | +- com.codahale.metrics:metrics-core:jar:3.0.1:compile
[INFO] | | +- com.codahale.metrics:metrics-ganglia:jar:3.0.1:compile
[INFO] | | | \- info.ganglia.gmetric4j:gmetric4j:jar:1.0.3:compile
[INFO] | | | \- org.acplt:oncrpc:jar:1.0.7:compile
[INFO] | | +- com.codahale.metrics:metrics-graphite:jar:3.0.1:compile
[INFO] | | +- com.spatial4j:spatial4j:jar:0.4.1:compile
[INFO] | | +- commons-collections:commons-collections:jar:3.2.1:compile
[INFO] | | +- commons-configuration:commons-configuration:jar:1.10:compile
[INFO] | | | +- commons-lang:commons-lang:jar:2.6:compile
[INFO] | | | \- commons-logging:commons-logging:jar:1.1.1:compile
[INFO] | | +- commons-codec:commons-codec:jar:1.7:compile
[INFO] | | +- com.google.guava:guava:jar:18.0:compile
[INFO] | | +- com.carrotsearch:hppc:jar:0.7.1:compile
[INFO] | | +- com.github.stephenc.high-scale-lib:high-scale-lib:jar:1.1.4:compile
[INFO] | | +- com.google.code.findbugs:jsr305:jar:3.0.0:compile
[INFO] | | \- org.slf4j:slf4j-api:jar:1.7.5:compile
[INFO] | +- com.thinkaurelius.titan:titan-berkeleyje:jar:1.0.0:compile
[INFO] | | \- com.sleepycat:je:jar:5.0.73:compile
[INFO] | +- com.thinkaurelius.titan:titan-es:jar:1.0.0:compile
[INFO] | | +- org.elasticsearch:elasticsearch:jar:1.5.1:compile
[INFO] | | | +- org.apache.lucene:lucene-core:jar:4.10.4:compile
[INFO] | | | +- org.apache.lucene:lucene-analyzers-common:jar:4.10.4:compile
[INFO] | | | +- org.apache.lucene:lucene-queries:jar:4.10.4:compile
[INFO] | | | +- org.apache.lucene:lucene-memory:jar:4.10.4:compile
[INFO] | | | +- org.apache.lucene:lucene-highlighter:jar:4.10.4:compile
[INFO] | | | +- org.apache.lucene:lucene-queryparser:jar:4.10.4:compile
[INFO] | | | +- org.apache.lucene:lucene-sandbox:jar:4.10.4:compile
[INFO] | | | +- org.apache.lucene:lucene-suggest:jar:4.10.4:compile
[INFO] | | | +- org.apache.lucene:lucene-misc:jar:4.10.4:compile
[INFO] | | | +- org.apache.lucene:lucene-join:jar:4.10.4:compile
[INFO] | | | +- org.apache.lucene:lucene-grouping:jar:4.10.4:compile
[INFO] | | | +- org.apache.lucene:lucene-spatial:jar:4.10.4:compile
[INFO] | | | +- org.ow2.asm:asm:jar:4.1:compile
[INFO] | | | \- org.ow2.asm:asm-commons:jar:4.1:compile
[INFO] | | \- org.antlr:antlr-runtime:jar:3.2:compile
[INFO] | | \- org.antlr:stringtemplate:jar:3.2:compile
[INFO] | | \- antlr:antlr:jar:2.7.7:compile
[INFO] | \- com.amazonaws:dynamodb-titan100-storage-backend:jar:1.0.0:compile
[INFO] | +- com.amazonaws:aws-java-sdk-dynamodb:jar:1.11.7:compile
[INFO] | | +- com.amazonaws:aws-java-sdk-s3:jar:1.11.7:compile
[INFO] | | | \- com.amazonaws:aws-java-sdk-kms:jar:1.11.7:compile
[INFO] | | \- com.amazonaws:aws-java-sdk-core:jar:1.11.7:compile
[INFO] | | +- org.apache.httpcomponents:httpclient:jar:4.5.2:compile
[INFO] | | | \- org.apache.httpcomponents:httpcore:jar:4.4.4:compile
[INFO] | | +- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.6.6:compile
[INFO] | | \- joda-time:joda-time:jar:2.8.1:compile
[INFO] | +- com.amazonaws:DynamoDBLocal:jar:1.11.0.1:compile
[INFO] | | +- org.antlr:antlr4-runtime:jar:4.1:compile
[INFO] | | | \- org.abego.treelayout:org.abego.treelayout.core:jar:1.0.1:compile
[INFO] | | +- commons-cli:commons-cli:jar:1.2:compile
[INFO] | | +- com.almworks.sqlite4java:libsqlite4java-linux-i386:so:1.0.392:runtime
[INFO] | | | \- com.almworks.sqlite4java:sqlite4java:jar:1.0.392:runtime
[INFO] | | +- com.almworks.sqlite4java:libsqlite4java-linux-amd64:so:1.0.392:runtime
[INFO] | | +- com.almworks.sqlite4java:sqlite4java-win32-x64:dll:1.0.392:runtime
[INFO] | | +- com.almworks.sqlite4java:sqlite4java-win32-x86:dll:1.0.392:runtime
[INFO] | | +- com.almworks.sqlite4java:libsqlite4java-osx:dylib:1.0.392:runtime
[INFO] | | +- org.apache.logging.log4j:log4j-api:jar:2.1:compile
[INFO] | | +- org.apache.logging.log4j:log4j-core:jar:2.1:compile
[INFO] | | +- org.eclipse.jetty:jetty-client:jar:8.1.12.v20130726:compile
[INFO] | | \- org.mockito:mockito-core:jar:1.10.19:compile
[INFO] | +- au.com.bytecode:opencsv:jar:2.4:compile
[INFO] | +- com.fasterxml.jackson.datatype:jackson-datatype-json-org:jar:2.5.3:compile
[INFO] | | +- com.fasterxml.jackson.core:jackson-core:jar:2.5.3:compile
[INFO] | | \- org.apache.geronimo.bundles:json:jar:20090211_1:compile
[INFO] | \- org.slf4j:slf4j-log4j12:jar:1.7.5:compile
[INFO] +- com.mohataher.xx:yy-graph-model:jar:1.0-SNAPSHOT:compile
[INFO] +- com.mohataher.xx:importer:jar:1.0-SNAPSHOT:compile
[INFO] | +- org.apache.xx:gremlin-core:jar:3.0.1-incubating:compile
[INFO] | | +- org.apache.xx:gremlin-shaded:jar:3.0.1-incubating:compile
[INFO] | | +- org.yaml:snakeyaml:jar:1.15:compile
[INFO] | | +- org.javatuples:javatuples:jar:1.2:compile
[INFO] | | +- com.fasterxml.jackson.core:jackson-databind:jar:2.5.3:compile
[INFO] | | | \- com.fasterxml.jackson.core:jackson-annotations:jar:2.5.0:compile
[INFO] | | +- com.jcabi:jcabi-manifests:jar:1.1:compile
[INFO] | | | \- com.jcabi:jcabi-log:jar:0.14:compile
[INFO] | | \- org.slf4j:jcl-over-slf4j:jar:1.7.12:compile
[INFO] | +- org.apache.xx:gremlin-groovy:jar:3.0.1-incubating:compile
[INFO] | | +- org.apache.ivy:ivy:jar:2.3.0:compile
[INFO] | | +- org.codehaus.groovy:groovy:jar:indy:2.4.1:compile
[INFO] | | +- org.codehaus.groovy:groovy-groovysh:jar:indy:2.4.1:compile
[INFO] | | | +- org.codehaus.groovy:groovy-console:jar:2.4.1:compile
[INFO] | | | | +- org.codehaus.groovy:groovy-templates:jar:2.4.1:compile
[INFO] | | | | | \- org.codehaus.groovy:groovy-xml:jar:2.4.1:compile
[INFO] | | | | \- org.codehaus.groovy:groovy-swing:jar:2.4.1:compile
[INFO] | | | +- jline:jline:jar:2.12:compile
[INFO] | | | \- org.codehaus.groovy:groovy:jar:2.4.1:compile
[INFO] | | +- org.codehaus.groovy:groovy-json:jar:indy:2.4.1:compile
[INFO] | | +- org.codehaus.groovy:groovy-jsr223:jar:indy:2.4.1:compile
[INFO] | | +- org.apache.commons:commons-lang3:jar:3.3.1:compile
[INFO] | | \- org.mindrot:jbcrypt:jar:0.3m:compile
[INFO] | +- org.apache.xx:tinkergraph-gremlin:jar:3.0.1-incubating:compile
[INFO] | +- junit:junit:jar:4.12:compile
[INFO] | | \- org.hamcrest:hamcrest-core:jar:1.3:compile
[INFO] | +- commons-io:commons-io:jar:2.4:compile
[INFO] | \- org.apache.spark:spark-core_2.10:jar:1.2.2:compile
[INFO] | +- com.twitter:chill_2.10:jar:0.5.0:compile
[INFO] | | \- com.esotericsoftware.kryo:kryo:jar:2.21:compile
[INFO] | | +- com.esotericsoftware.reflectasm:reflectasm:jar:shaded:1.07:compile
[INFO] | | +- com.esotericsoftware.minlog:minlog:jar:1.2:compile
[INFO] | | \- org.objenesis:objenesis:jar:1.2:compile
[INFO] | +- com.twitter:chill-java:jar:0.5.0:compile
[INFO] | +- org.apache.hadoop:hadoop-client:jar:2.2.0:compile
[INFO] | | +- org.apache.hadoop:hadoop-common:jar:2.2.0:compile
[INFO] | | | +- org.apache.commons:commons-math:jar:2.1:compile
[INFO] | | | +- xmlenc:xmlenc:jar:0.52:compile
[INFO] | | | +- org.codehaus.jackson:jackson-core-asl:jar:1.8.8:compile
[INFO] | | | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:compile
[INFO] | | | +- org.apache.avro:avro:jar:1.7.4:compile
[INFO] | | | +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO] | | | +- org.apache.hadoop:hadoop-auth:jar:2.2.0:compile
[INFO] | | | \- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO] | | | \- org.tukaani:xz:jar:1.0:compile
[INFO] | | +- org.apache.hadoop:hadoop-hdfs:jar:2.2.0:compile
[INFO] | | | \- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO] | | +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.2.0:compile
[INFO] | | | +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.2.0:compile
[INFO] | | | | +- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:compile
[INFO] | | | | \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.2.0:compile
[INFO] | | | \- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.2.0:compile
[INFO] | | +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:compile
[INFO] | | +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.2.0:compile
[INFO] | | | \- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:compile
[INFO] | | +- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.2.0:compile
[INFO] | | \- org.apache.hadoop:hadoop-annotations:jar:2.2.0:compile
[INFO] | +- org.apache.spark:spark-network-common_2.10:jar:1.2.2:compile
[INFO] | +- org.apache.spark:spark-network-shuffle_2.10:jar:1.2.2:compile
[INFO] | +- net.java.dev.jets3t:jets3t:jar:0.7.1:compile
[INFO] | | \- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO] | +- org.apache.curator:curator-recipes:jar:2.4.0:compile
[INFO] | | +- org.apache.curator:curator-framework:jar:2.4.0:compile
[INFO] | | | \- org.apache.curator:curator-client:jar:2.4.0:compile
[INFO] | | \- org.apache.zookeeper:zookeeper:jar:3.4.5:compile
[INFO] | +- org.eclipse.jetty:jetty-plus:jar:8.1.14.v20131031:compile
[INFO] | | +- org.eclipse.jetty.orbit:javax.transaction:jar:1.1.1.v201105210645:compile
[INFO] | | +- org.eclipse.jetty:jetty-webapp:jar:8.1.14.v20131031:compile
[INFO] | | | +- org.eclipse.jetty:jetty-xml:jar:8.1.14.v20131031:compile
[INFO] | | | \- org.eclipse.jetty:jetty-servlet:jar:8.1.14.v20131031:compile
[INFO] | | \- org.eclipse.jetty:jetty-jndi:jar:8.1.14.v20131031:compile
[INFO] | | \- org.eclipse.jetty.orbit:javax.mail.glassfish:jar:1.4.1.v201005082020:compile
[INFO] | | \- org.eclipse.jetty.orbit:javax.activation:jar:1.1.0.v201105071233:compile
[INFO] | +- org.eclipse.jetty:jetty-security:jar:8.1.14.v20131031:compile
[INFO] | +- org.eclipse.jetty:jetty-util:jar:8.1.14.v20131031:compile
[INFO] | +- org.eclipse.jetty:jetty-server:jar:8.1.14.v20131031:compile
[INFO] | | +- org.eclipse.jetty.orbit:javax.servlet:jar:3.0.0.v201112011016:compile
[INFO] | | +- org.eclipse.jetty:jetty-continuation:jar:8.1.14.v20131031:compile
[INFO] | | \- org.eclipse.jetty:jetty-http:jar:8.1.14.v20131031:compile
[INFO] | | \- org.eclipse.jetty:jetty-io:jar:8.1.14.v20131031:compile
[INFO] | +- org.apache.commons:commons-math3:jar:3.1.1:compile
[INFO] | +- org.slf4j:jul-to-slf4j:jar:1.7.5:compile
[INFO] | +- com.ning:compress-lzf:jar:1.0.0:compile
[INFO] | +- org.xerial.snappy:snappy-java:jar:1.1.1.6:compile
[INFO] | +- net.jpountz.lz4:lz4:jar:1.2.0:compile
[INFO] | +- org.roaringbitmap:RoaringBitmap:jar:0.4.5:compile
[INFO] | +- commons-net:commons-net:jar:2.2:compile
[INFO] | +- org.spark-project.akka:akka-remote_2.10:jar:2.3.4-spark:compile
[INFO] | | +- org.spark-project.akka:akka-actor_2.10:jar:2.3.4-spark:compile
[INFO] | | | \- com.typesafe:config:jar:1.2.1:compile
[INFO] | | +- io.netty:netty:jar:3.8.0.Final:compile
[INFO] | | +- org.spark-project.protobuf:protobuf-java:jar:2.5.0-spark:compile
[INFO] | | \- org.uncommons.maths:uncommons-maths:jar:1.2.2a:compile
[INFO] | +- org.spark-project.akka:akka-slf4j_2.10:jar:2.3.4-spark:compile
[INFO] | +- org.scala-lang:scala-library:jar:2.10.4:compile
[INFO] | +- org.json4s:json4s-jackson_2.10:jar:3.2.10:compile
[INFO] | | \- org.json4s:json4s-core_2.10:jar:3.2.10:compile
[INFO] | | +- org.json4s:json4s-ast_2.10:jar:3.2.10:compile
[INFO] | | +- com.thoughtworks.paranamer:paranamer:jar:2.6:compile
[INFO] | | \- org.scala-lang:scalap:jar:2.10.0:compile
[INFO] | | \- org.scala-lang:scala-compiler:jar:2.10.0:compile
[INFO] | | \- org.scala-lang:scala-reflect:jar:2.10.0:compile
[INFO] | +- org.apache.mesos:mesos:jar:shaded-protobuf:0.18.1:compile
[INFO] | +- io.netty:netty-all:jar:4.0.23.Final:compile
[INFO] | +- com.clearspring.analytics:stream:jar:2.7.0:compile
[INFO] | +- com.codahale.metrics:metrics-jvm:jar:3.0.0:compile
[INFO] | +- com.codahale.metrics:metrics-json:jar:3.0.0:compile
[INFO] | +- org.tachyonproject:tachyon-client:jar:0.5.0:compile
[INFO] | | \- org.tachyonproject:tachyon:jar:0.5.0:compile
[INFO] | +- org.spark-project:pyrolite:jar:2.0.1:compile
[INFO] | +- net.sf.py4j:py4j:jar:0.8.2.1:compile
[INFO] | \- org.spark-project.spark:unused:jar:1.0.0:compile
[INFO] +- org.bayofmany.peapod:peapod:jar:0.2.2:compile
[INFO] | +- com.squareup:javawriter:jar:2.5.1:compile
[INFO] | \- org.reflections:reflections:jar:0.9.10:compile
[INFO] | +- org.javassist:javassist:jar:3.19.0-GA:compile
[INFO] | \- com.google.code.findbugs:annotations:jar:2.0.1:compile
[INFO] \- log4j:log4j:jar:1.2.17:compile
Why is this happening?
The code structure was not the issue. The problem was caused by Spark 1.2.2. When I run the code using spark-submit, the SparkSubmit.java loads an older version of httpclient assigned to Spark code and that causes this conflict.
The solution I used was simple, I used apache-shadeplugin`, shaded the library and it worked quite well.
This means that both versions will work simultaneously without any issues.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<relocations>
<relocation>
<pattern>org.apache.http</pattern>
<shadedPattern>shaded.org.apache.http</shadedPattern>
</relocation>
</relocations>
......