Search code examples
javamavenintellij-ideaapache-flink

Maven build has missing package linkages

I am having problems using Maven. I have an Apache Flink project and wanted to run it on my server. Locally it runs fine but on the server it aborts with the error:

java.lang.NoClassDefFoundError: org/apache/flink/examples/java/ml/util/LinearRegressionData

In my Java project I imported the class with

import org.apache.flink.examples.java.ml.util.LinearRegressionData;

And I used the correct class at the import:

enter image description here

After the build I had a look into the Jar file. The following classes were included:

enter image description here

The /util/ folder is completely missing. My dependency-section in the pom file looks like the following:

<dependencies>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-java</artifactId>
        <version>0.9.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-streaming-core</artifactId>
        <version>0.9.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-clients</artifactId>
        <version>0.9.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-table</artifactId>
        <version>0.9.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-runtime</artifactId>
        <version>0.9.0</version>
    </dependency>
</dependencies>

When I have seen the package organization in the repository located at https://github.com/apache/flink/tree/release-0.9, I thought it would be possible to add the following lines to flink:

    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-examples</artifactId>
        <version>0.9.0</version>
    </dependency>

But these dependencies cannot be resolved. Since Maven does not throw an error when performing a clean install, I think this is a dependency issue. I thought Maven would include all used imports automatically. How can I make this runnable on my server?

Solution

  • You should include ML example as follows:

     <dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-java-examples</artifactId>
    <version>0.9.0</version>
</dependency>

    flink-examples is a parent pom module; not a jar module. How do you build your jar file? Using maven-jar-plugin? A regular mvn package or mvn install does not packages dependencies.

    Using maven-jar-plugin requires to specify what you need to include using <include>. See here: https://maven.apache.org/plugins/maven-jar-plugin/examples/include-exclude.html

    In your case it should be something like this:

    <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-jar-plugin</artifactId>
    <configuration>
        <archive>
            <manifestEntries>
                <Main-Class>org.apache.flink.examples.java.ml.LinearRegression</Main-Class>
            </manifestEntries>
        </archive>

        <includes>
                <include>org/apache/flink/examples/java/ml/*.class</include>
                <include>org/apache/flink/examples/java/ml/util/*.class</include>
        </includes>
    </configuration>
</plugin>

    You can also compare with https://github.com/apache/flink/blob/master/flink-examples/flink-java-examples/pom.xml

    You also need to "pull" and unpack your dependencies into your project such that they can be re-packages using maven-dependency-plugin

    <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-dependency-plugin</artifactId>
    <version>2.9</version><!--$NO-MVN-MAN-VER$-->
    <executions>
        <execution>
            <id>unpack</id>
            <phase>prepare-package</phase>
            <goals>
                <goal>unpack</goal>
            </goals>
            <configuration>
                <artifactItems>
                    <artifactItem>
                        <groupId>org.apache.flink</groupId>
                        <artifactId>flink-java-examples</artifactId>
                        <version>0.9.0</version>
                        <type>jar</type>
                        <overWrite>false</overWrite>
                        <outputDirectory>${project.build.directory}/classes</outputDirectory>
                    </artifactItem>
                </artifactItems>
            </configuration>
        </execution>
    </executions>
</plugin>