Search code examples
mavenhadoopcloudera

Maven - Different Dependency Version in Test

I'm suffering an issue similar to Maven 2 - different dependency versions in test and compile but the specified answer there does not work.

In my project I need to depend on a Cloudera distribution of Hadoop and a 'vanilla' version for JUnit testing, as the former only works on *nix.

When I try and execute my application, I get Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration. When I run JUnit tests from Maven or Eclipse, everything works fine. If I comment out the test dependencies, the application runs successfully.

Why is the compile dependency getting ignored when the test dependency is uncommented?

    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-core</artifactId>
        <version>0.20.2-cdh3u2</version>
        <scope>compile</scope>
    </dependency>

    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-core</artifactId>
        <version>1.0.0</version>
        <scope>test</scope>
    </dependency>

    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-test</artifactId>
        <version>1.0.0</version>
        <scope>test</scope>
    </dependency>

mvn dependency:list is shown below. The compile scoped version does not show up at all:

[INFO] The following files have been resolved:
[INFO]    ant:ant:jar:1.6.5:test
[INFO]    aopalliance:aopalliance:jar:1.0:compile
[INFO]    asm:asm:jar:3.3.1:compile
[INFO]    cglib:cglib:jar:2.2.2:compile
[INFO]    ch.qos.logback:logback-classic:jar:1.0.0:compile
[INFO]    ch.qos.logback:logback-core:jar:1.0.0:compile
[INFO]    com.google.guava:guava:jar:r08:compile
[INFO]    com.h2database:h2:jar:1.3.164:test
[INFO]    com.jolbox:bonecp:jar:0.7.1.RELEASE:compile
[INFO]    com.sun.jersey:jersey-core:jar:1.11:test
[INFO]    commons-beanutils:commons-beanutils:jar:1.7.0:test
[INFO]    commons-beanutils:commons-beanutils-core:jar:1.8.0:test
[INFO]    commons-cli:commons-cli:jar:1.2:test
[INFO]    commons-codec:commons-codec:jar:1.4:test
[INFO]    commons-collections:commons-collections:jar:3.2.1:test
[INFO]    commons-configuration:commons-configuration:jar:1.6:test
[INFO]    commons-digester:commons-digester:jar:1.8:test
[INFO]    commons-el:commons-el:jar:1.0:test
[INFO]    commons-httpclient:commons-httpclient:jar:3.0.1:test
[INFO]    commons-lang:commons-lang:jar:2.4:test
[INFO]    commons-logging:commons-logging:jar:1.1.1:compile
[INFO]    commons-net:commons-net:jar:1.4.1:test
[INFO]    hsqldb:hsqldb:jar:1.8.0.10:test
[INFO]    junit:junit:jar:4.10:test
[INFO]    mysql:mysql-connector-java:jar:5.1.18:compile
[INFO]    net.java.dev.jets3t:jets3t:jar:0.7.1:test
[INFO]    net.sf.kosmosfs:kfs:jar:0.3:test
[INFO]    org.apache.commons:commons-math:jar:2.1:test
[INFO]    org.apache.ftpserver:ftplet-api:jar:1.0.0:test
[INFO]    org.apache.ftpserver:ftpserver-core:jar:1.0.0:test
[INFO]    org.apache.ftpserver:ftpserver-deprecated:jar:1.0.0-M2:test
[INFO]    org.apache.hadoop:hadoop-core:jar:1.0.0:test
[INFO]    org.apache.hadoop:hadoop-test:jar:1.0.0:test
[INFO]    org.apache.mina:mina-core:jar:2.0.0-M5:test
[INFO]    org.codehaus.jackson:jackson-core-asl:jar:1.0.1:test
[INFO]    org.codehaus.jackson:jackson-mapper-asl:jar:1.0.1:test
[INFO]    org.eclipse.jdt:core:jar:3.1.1:test
[INFO]    org.hamcrest:hamcrest-core:jar:1.1:test
[INFO]    org.liquibase:liquibase-core:jar:2.0.3:test
[INFO]    org.liquibase.ext:liquibase-slf4j:jar:0.0.1:test
[INFO]    org.mortbay.jetty:jetty:jar:6.1.26:test
[INFO]    org.mortbay.jetty:jetty-util:jar:6.1.26:test
[INFO]    org.mortbay.jetty:jsp-2.1:jar:6.1.14:test
[INFO]    org.mortbay.jetty:jsp-api-2.1:jar:6.1.14:test
[INFO]    org.mortbay.jetty:servlet-api:jar:2.5-20081211:test
[INFO]    org.mortbay.jetty:servlet-api-2.5:jar:6.1.14:test
[INFO]    org.slf4j:jcl-over-slf4j:jar:1.6.4:compile
[INFO]    org.slf4j:log4j-over-slf4j:jar:1.6.4:compile
[INFO]    org.slf4j:slf4j-api:jar:1.6.4:compile
[INFO]    org.springframework:spring-aop:jar:3.1.1.RELEASE:compile
[INFO]    org.springframework:spring-asm:jar:3.1.1.RELEASE:compile
[INFO]    org.springframework:spring-beans:jar:3.1.1.RELEASE:compile
[INFO]    org.springframework:spring-context:jar:3.1.1.RELEASE:compile
[INFO]    org.springframework:spring-context-support:jar:3.1.1.RELEASE:compile
[INFO]    org.springframework:spring-core:jar:3.1.1.RELEASE:compile
[INFO]    org.springframework:spring-expression:jar:3.1.1.RELEASE:compile
[INFO]    org.springframework:spring-jdbc:jar:3.1.1.RELEASE:compile
[INFO]    org.springframework:spring-test:jar:3.1.1.RELEASE:test
[INFO]    org.springframework:spring-tx:jar:3.1.1.RELEASE:compile
[INFO]    org.springframework.data:spring-data-hadoop:jar:1.0.0.BUILD-SNAPSHOT:c
ompile
[INFO]    oro:oro:jar:2.0.8:test
[INFO]    tomcat:jasper-compiler:jar:5.5.12:test
[INFO]    tomcat:jasper-runtime:jar:5.5.12:test
[INFO]    xmlenc:xmlenc:jar:0.52:test
Solution

  • Looks like a maven bug - or if not, something that is fishy enough I wouldn't trust the documentation if there were any.

    In your case, I would probably:

    1. upgrade maven and see if the issue is fixed in the latest m3
    2. try repackaging one of the hadoop versions with another artifact id or group name. maven-shade-plugin, don't bother changing the package name, but just introduce something maven won't know is no different from org.apache.hadoop:hadoop-core
    3. try moving the tests to a separate project downstream
    4. start debugging maven and understand what's going on exactly (and then fall back on 1 or 2...)