Search code examples
mavenintellij-ideapom.xmlapache-storm

how to build latest STORM source code to use core/external jars as dependencies used in application dev POM.xml


plan to develop a storm application to use kafka/mongodb, only latest STORM on github master support statequery of mongodb, so i have to build STORM myself to use it. Success to build STORM distribution package. But also i need to build jars which are used in application POM.xml. i used "mvn install" in STORM root project, and all STORM jars are successfully built and copied to my local maven repository. But when it comes to build application, dependencies can be resolved in POM. But all symbols of class/functions of those jar included in dependencies can'be resolved, all underscore lined in red color.

Anyone who have built latest STORM source code can help me on this?

These are key parts of my POM.xml:

    <groupId>com.fm.data</groupId>
    <artifactId>fmstreaming</artifactId>
    <version>1.0-SNAPSHOT</version>
    <properties>
        <storm.version>2.0.0-SNAPSHOT</storm.version>

        <storm.kafka.client.version>0.10.1.0</storm.kafka.client.version>
        <hbase.version>1.2.0</hbase.version>
        <mongodb.version>3.4.2</mongodb.version>

        <java.version>1.8</java.version>
        <java.encoding>UTF-8</java.encoding>

        <project-sourceEncoding>UTF-8</project-sourceEncoding>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <maven.compiler.encoding>UTF-8</maven.compiler.encoding>

    </properties>

    <repositories>
        <repository>
            <id>nexus</id>
            <name>nexus</name>
            <url>http://pro-hbase01:8081/nexus/content/groups/public/</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>

    </repositories>

    <dependencies>
        <dependency>
            <groupId>org.apache.storm</groupId>
            <artifactId>storm-core</artifactId>
            <version>${storm.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.storm</groupId>
            <artifactId>storm-kafka-client</artifactId>
            <version>${storm.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>${storm.kafka.client.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.storm</groupId>
            <artifactId>storm-mongodb</artifactId>
            <version>${storm.version}</version>
        </dependency>

    </dependencies>

    <build>
        <outputDirectory>target\classes</outputDirectory>
<!--
        <testOutputDirectory>target\test-classes</testOutputDirectory>
-->
        <testOutputDirectory>target\classes</testOutputDirectory>
        <resources>
            <resource>
                <directory>${project.basedir}/src/main/resources</directory>
                <filtering>true</filtering>
            </resource>
        </resources>
        <plugins>
            <plugin>
                    <artifactId>maven-compiler-plugin</artifactId>
                    <configuration>
                        <source>1.8</source>
                        <target>1.8</target>
                        <encoding>UTF-8</encoding>
                    </configuration>
                </plugin>

            <plugin>
                <groupId>org.codehaus.mojo </groupId>
                <artifactId>build-helper-maven-plugin </artifactId>
                <version>1.9.1 </version>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-eclipse-plugin</artifactId>
                <configuration>
                    <downloadSources>true</downloadSources>
                    <buildcommands>
                        <buildcommand>ch.epfl.lamp.sdt.core.scalabuilder</buildcommand>
                    </buildcommands>
                    <additionalProjectnatures>
                        <projectnature>ch.epfl.lamp.sdt.core.scalanature</projectnature>
                    </additionalProjectnatures>
                    <classpathContainers>
                        <classpathContainer>org.eclipse.jdt.launching.JRE_CONTAINER</classpathContainer>
                        <classpathContainer>ch.epfl.lamp.sdt.launching.SCALA_CONTAINER</classpathContainer>
                    </classpathContainers>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
                <version>2.2</version>
                <executions>
                    <execution>
                        <id>copy-dependencies</id>
                        <phase>package</phase>
                        <goals>
                            <goal>copy-dependencies</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>${project.build.directory}/lib</outputDirectory>
                            <overWriteReleases>false</overWriteReleases>
                            <overWriteSnapshots>false</overWriteSnapshots>
                            <overWriteIfNewer>true</overWriteIfNewer>
                        </configuration>
                    </execution>
                </executions>
            </plugin>


            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.5</version>
                <configuration>
                    <testFailureIgnore>true</testFailureIgnore>
                </configuration>
            </plugin>

        </plugins>
    </build>
    <reporting>
        <plugins>
        </plugins>
    </reporting>

Solution

  • If your local Maven repository contains the Storm jars (in version 2.0.0-SNAPSHOT), your pom should work. Maybe try running Maven from the command line (mvn install) on your project. That will let you rule out whether the issue is with your POM or with some IntelliJ setting.

    Also consider removing the Eclipse plugin from your POM, the plugin is deprecated and you really shouldn't need it in IntelliJ or newer Eclipse versions. Maybe IntelliJ is getting confused by the Eclipse project files?