Apache Storm - How to exclude storm jar in Maven in production cluster

As per the documentation, when we deploy Storm topology to production cluster, we have to exclude from Maven the jar of Storm, because it will be in the classpath already, and if we don't do this, there will be an error like multiple default.yaml in classpath.

So how can we do it? The documentation has provided some details but not clear enough. When we configure the maven and build the jar, the org.apache.storm jar is still included, we have a default.yaml in the jar, and, actually, if we turn on the debug output in building, we will see a warning like:

[WARNING]The following patterns were never triggered in this artifact inclusion filter: o 'org.apache.storm:storm-core:jar:1.1.1'.

Changed the scope of the dependency org.apache.storm to provided, and it does not work.

<dependency>
    <groupId>org.apache.storm</groupId>
    <artifactId>storm-core</artifactId>
    <type>jar</type>
    <version>1.1.1</version>
    <scope>provided</scope>
</dependency>

Solution

The documentation of Storm tells us to exclude the Storm dependency and provides a link to a page of Maven, which explains how to do it but it lacks a full example. It turns out that we have to create another XML file as the descriptor of assembly configuration, instead of putting the configuration directly in the pom.xml, even when Eclipse does not complain about putting the two files together.

And, in the second xml file, the scope should be compile instead of runtime.

Here is how:

We have our pom.xml with assembly plugins, pointing to another descriptor xml file:

<plugin>                                                                                
    <artifactId>maven-assembly-plugin</artifactId>                                      
    <executions>                                                                        
        <execution>                                                                     
            <id>make-assembly</id> <!-- this is used for inheritance merges -->         
            <phase>package</phase> <!-- bind to the packaging phase -->                 
            <goals>                                                                     
                <goal>single</goal>                                                     
            </goals>                                                                    
        </execution>                                                                    
    </executions>                                                                       
    <configuration>                                                                     
        <descriptors>                                                                   
            <descriptor>/src/main/resources/exclude-storm.xml</descriptor>              
        </descriptors>                                                                  
        <archive>                                                                       
            <manifest>                                                                  
                <mainClass>path.to.main.class</mainClass> 
            </manifest>                                                                 
        </archive>                                                                      
        <descriptorRefs>                                                                
            <descriptorRef>jar-with-dependencies</descriptorRef>                        
        </descriptorRefs>                                                               
    </configuration>                                                                    
</plugin>

Note the part of: (should be complete path)

<descriptors>                                                                   
    <descriptor>/src/main/resources/exclude-storm.xml</descriptor>              
</descriptors>

And in this path:

<?xml version="1.0" encoding="UTF-8"?>
<assembly xmlns="http://maven.apache.org/ASSEMBLY/2.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/ASSEMBLY/2.0.0 http://maven.apache.org/xsd/assembly-2.0.0.xsd">
    <id>exclude-storm</id>
    <formats>
        <format>jar</format>
    </formats>
    <includeBaseDirectory>false</includeBaseDirectory>
    <dependencySets>
        <dependencySet>
            <outputDirectory>/</outputDirectory>
            <useProjectArtifact>false</useProjectArtifact>
            <unpack>true</unpack>
            <scope>compile</scope>   <!-- note here!!!! -->
            <excludes>
                <exclude>org.apache.storm:storm-core:jar:1.1.1</exclude>
            </excludes>
        </dependencySet>
    </dependencySets>
    <fileSets>
        <fileSet>
            <outputDirectory>/</outputDirectory>
            <directory>${project.build.outputDirectory}</directory>
        </fileSet>
    </fileSets>
</assembly>

Here we add the settings of Maven doc page.

The most important thing: format of exclusion: should be: (ref)

groupId:artifactId:type[:classifier]:version

And the scope! runtime is default, I changed it to compile and it works.

At last, when you compile, use:

clean assembly:assembly

And turn on debug output to see full output in console. If you:

have a successful build
search the output and haven't found anything like: [WARNING]The following patterns were never triggered in this artifact inclusion filter: o 'org.apache.storm:storm-core:jar:1.1.1'.
the jar contains no default.yaml

Then you know you have succeeded.

Thanks for the inspiration of another question and answers: How to exclude dependencies from maven assembly plugin : jar-with-dependencies?