Search code examples
javamavenlog4jlog4j-1.2-cve

How to quickly detect and remove log4j classes from our code base and the base image? "mvn dependency:tree" does not check base image

We are building an app based on Red Hat JBoss AMQ 6. We wrap some Java code around the base image to provide extra functionalities which are lacking in AMQ 6.

Now, when the CVE of Log4j stroke, we found that this component is vulnerable because it uses log4j 1.x. Now I don't just mean our Java code uses it, but also the base image of Red Hat AMQ 6 uses it. As AMQ 6 is EOL now, Red Hat does not provide support anymore, so there will not be official releases with fix.

So, how do I remove vulnerable log4j 1.x classes from:

  • my Java code
  • the base image

?

I am building with jib maven plugin.

It seems mvn dependency:tree only gives info about our Java wrapper code, not about the base image. And I don't understand what does the "+" and "-" mean while it gives output like this:

[INFO] +- org.jboss.resteasy:resteasy-jaxrs:jar:3.7.0.Final:compile
[INFO] |  +- org.jboss.spec.javax.ws.rs:jboss-jaxrs-api_2.1_spec:jar:1.0.2.Final:compile
[INFO] |  +- org.jboss.spec.javax.xml.bind:jboss-jaxb-api_2.3_spec:jar:1.0.1.Final:compile
[INFO] |  +- org.reactivestreams:reactive-streams:jar:1.0.2:compile
[INFO] |  +- javax.validation:validation-api:jar:2.0.1.Final:compile
[INFO] |  +- org.jboss.spec.javax.annotation:jboss-annotations-api_1.3_spec:jar:1.0.1.Final:compile
[INFO] |  +- javax.activation:activation:jar:1.1.1:compile
[INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.5.4:compile
[INFO] |  |  +- org.apache.httpcomponents:httpcore:jar:4.4.7:compile
[INFO] |  |  +- commons-logging:commons-logging:jar:1.2:compile
[INFO] |  |  \- commons-codec:commons-codec:jar:1.10:compile
[INFO] |  +- commons-io:commons-io:jar:2.5:compile
[INFO] |  +- net.jcip:jcip-annotations:jar:1.0:compile
[INFO] |  \- org.jboss.logging:jboss-logging:jar:3.3.2.Final:compile

Does + mean that it can be expanded further but is not shown here?

Some background here: https://nvd.nist.gov/vuln/detail/CVE-2021-44228

Solution

  • Step 1: Dealing with our wrapping Java code base

    For our code, we do:

    mvn dependency:tree | grep log4j

    And we found some dep from other teams bringing in transitive log4j 1.17. Informed that team and they fixed that in a recent version, we just change in our pom the version to be it, and our pom is fixed.

    If your dependency is not maintained anymore, you can enter the artifactory of your organization, and manually look for the classes below in all jars you need(the list is long, because apart from CVE-2021-4104 which mentions JMSAppender, I found log4j 1.x has a lot of other vulnerabilities and more classes should be removed)

    • org/apache/log4j/net/SocketServer.class
    • org/apache/log4j/net/SimpleSocketServer.class (just in case)
    • org/apache/log4j/net/SocketAppender.class
    • org/apache/log4j/net/SMTPAppender$1.class
    • org/apache/log4j/net/SMTPAppender.class
    • org/apache/log4j/net/JMSAppender.class
    • org/apache/log4j/net/JMSSink.class
    • org/apache/log4j/net/JDBCAppender.class
    • org/apache/log4j/chainsaw/*.class

    If you cannot fix your internal Nexus repo/artifactory, you can find the jar of log4j in local Maven registry (under ~/.m2) and remove the class; then you build your app again; but remember don't use -U to redownload the jar from remote registry.

    Step 2: Dealing with base image jars

    To find other libs in the base image containing log4j is more complicated.

    • Tampering the layers by removing the classes files cannot go undetected by Docker daemon. The sha256 value changes, you have to replace the sha256 value in the json file in the main dir with new sha256sum layer.tar; but even with that, Docker daemon will give error when you load the tar: Cannot open /var/lib/docker/tmp-xxxx/...: file not found or so.

    • Then I tried to create a script to remove the classes at runtime, right before running the app, and define a new entrypoint in jib to run it before running the app.

    #!/bin/sh
/opt/amq/bin/fix_log4j_1.x_cves.sh
/opt/amq/bin/launch.sh # the original, inherited entrypoint in jib

    But then I found it will slow down pod startup; unresponsive pods may be restarted by Openshift, causing unwanted delay and errors. But the output of this script gives me an idea of which jars contain the classes to remove, which is a solid basis for my next solution.

    At last, I came up with a perfect solution

    1. Implement the previous solution, docker run the image, and note down the jars' names in the output of the script.
    Starting to fix all CVEs regarding Log4j 1.x...
>>>>> Removing class file from '/opt/amq/lib/optional/log4j-1.2.17.redhat-1.jar':
removed 'org/apache/log4j/chainsaw/ControlPanel$1.class'
removed 'org/apache/log4j/chainsaw/ControlPanel$2.class'
...
>>>>> Removing class file from '/opt/amq/activemq-all-5.11.0.redhat-630495.jar':
...
    1. Define these two as provided dependencies in pom.xml
    2. Use maven copy-dependency plugin to copy them into build folder (by default target/dependency)
    3. Use maven exec plugin to run the same script against target/dependency dir to remove vulnerable classes, while building the image with jib
    4. Use maven jib plugin to copy the fixed jars into the container, so that they will be on a new layer on top of all previous layer, to shadow/whiteout the unfixed jar (see answer to my other question)

    By doing this, we eliminate the vulnerable classes while building the image, pod startup speed is not compromised, and the binary transferred to production image registry is already safe.

    An advantage of this approach is we are not limited by available tools provided by the container because the script runs now in our local environment. We can install whatever tool we need and use them in the script. For example, in the original script I defined function extract_remove_repackage to complete a simple task of extracting+remove classes+repackaging, only because zip is not installed in the base image. But in my local machine, this can be done by zip in one line,

    You have to make sure to bind 3, 4 and 5 to different maven build phases, so that they happen in such order. I bind 3) to compile, 4) to process-classes and 5) to package.

    Implementation details below:

    • The script (in my previous solution, put under src/main/jib/opt/amq/bin, so it could be copied into container. Also you needed the new entrypoint script here in the same folder. Now in this solution, moved to src/main/scripts)

    fix_log4j_1.x_cves.sh:

    #!/bin/bash
# Script to fix log4j 1.x CVEs. Initially it is only for CVE-2021-4104, but
# since there are multiple CVEs regarding log4j 1.x, they are all fixed here:

# Class File                                        CVE
# org/apache/log4j/net/SocketAppender.class         CVE-2019-17571
# org/apache/log4j/net/SocketServer.class           CVE-2019-17571
# org/apache/log4j/net/SMTPAppender$1.class         CVE-2020-9488
# org/apache/log4j/net/SMTPAppender.class           CVE-2020-9488
# org/apache/log4j/net/JMSAppender.class            CVE-2021-4104
# org/apache/log4j/net/JMSSink.class                CVE-2022-23302
# org/apache/log4j/net/JDBCAppender.class           CVE-2022-23305
# org/apache/log4j/chainsaw/*.class                 CVE-2022-23307

cves=(
'CVE-2019-17571'
'CVE-2019-17571'
'CVE-2020-9488'
'CVE-2020-9488'
'CVE-2021-4104'
'CVE-2022-23302'
'CVE-2022-23305'
'CVE-2022-23307'
)

size() {
    stat -c %s "$1"
}

extract_remove_repackage() {
    before=$1
    # jar xf -C some_dir only extract to current dir, we have to cd first
    jar_dir=$(dirname "$2")
    jar_file=$(basename "$2")
    temp_dir=$jar_dir/temp
    mkdir "$temp_dir"
    cp list.txt "$temp_dir"/ && cp "$2" "$temp_dir"/
    cd "$temp_dir"
    jar xf "$jar_file"
    # provide file and dir names to rm with list.txt
    xargs rm -rvf < list.txt && rm list.txt "$jar_file"
    jar cf "$jar_file" .
    mv "$jar_file" ../
    # go back and clean up
    cd "$before" && rm -rf "$temp_dir"
}

find_vulnerable_jars() {
    cd "$root_dir"
    jar -tvf "$1" | grep -E "$pattern" | awk '{ print $8 }' > list.txt
    if [ "$(size list.txt)" -gt 0 ]; then
        echo ">>>>> Removing class file from '$(realpath "$1")'":
        extract_remove_repackage "$(pwd)" "$1"
    else
        return 0
    fi
}
remove_classes_from_jars() {
    echo Starting to fix all CVEs regarding Log4j 1.x...
    # exclude jolokia.jar(link)
    # xargs can return error level to "if", when any of execution fails, while "find -exec" cannot
    # because we use custom function, xargs needs "bash -c"; thus we have to use "_" to pass each arg
    if find "$root_dir" -name "*.jar" -not -type l -print0 | xargs -0 -n1 bash -c 'find_vulnerable_jars "$@"' _; then
        echo All vunerable classes removed. CVE addressed:
        printf '%s\n' "${cves[@]}"
    else
        echo "Error while removing classes; exiting..."
        return 1
    fi
}

# to be able to use in find -exec child shell, we need to export all vars and functions
# $1: where to search jars, should match copy-dependency output dir.
export root_dir=$1
export pattern=".*(JMS|JDBC|SMTP|Socket)Appender.*.class|.*SocketServer.class|.*JMSSink.class|org/apache/log4j/chainsaw/.*"
export -f size
export -f extract_remove_repackage
export -f find_vulnerable_jars
remove_classes_from_jars
    • Define provided dependencies:
    <dependency>
    <groupId>org.apache.activemq</groupId>
    <artifactId>activemq-all</artifactId>
    <version>${version.activemq-all}</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>log4j</groupId>
    <artifactId>log4j</artifactId>
    <version>${version.log4j}</version>
    <scope>provided</scope>
</dependency>
    • copy-dependency plugin
    <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-dependency-plugin</artifactId>
    <executions>
        <execution>
            <id>copy-cve-jars</id>
            <phase>compile</phase>
            <goals>
                <goal>copy-dependencies</goal>
            </goals>
            <configuration>
                <includeArtifactIds>activemq-all,log4j</includeArtifactIds>
                <includeScope>provided</includeScope>
                <includeTypes>jar</includeTypes>
                <outputDirectory>${project.build.directory}/dependency</outputDirectory> <!-- default value -->
                <excludeTransitive>true</excludeTransitive>
            </configuration>
        </execution>
    </executions>
</plugin>
    • exec-plugin:
    <plugin>
    <artifactId>exec-maven-plugin</artifactId>
    <groupId>org.codehaus.mojo</groupId>
    <executions>
        <execution>
            <id>remove-cve-classes</id>
            <phase>process-classes</phase>
            <goals>
                <goal>exec</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <executable>${project.build.scriptSourceDirectory}/log4j_cve_fix.sh</executable>
        <arguments>
            <!-- should match copy-dependency output dir -->
            <argument>${project.build.directory}/dependency</argument>
        </arguments>
    </configuration>
</plugin>

    jib plugin: (needs to be > 3.0.0 to be able to use <path><inclueds>)

    <plugin>
    <groupId>com.google.cloud.tools</groupId>
    <artifactId>jib-maven-plugin</artifactId>
    <configuration>
        <from>
            <image>${docker.base.image}</image>
        </from>
        <to>
            <image>${docker.image}</image>
            <tags>
                <tag>${project.version}</tag>
            </tags>
        </to>
        <container>
            <appRoot>/dev/null</appRoot>
            <entrypoint>INHERIT</entrypoint> <!-- customized entrypoint not needed anymore, just revert to the way it was -->
        </container>
        <containerizingMode>packaged</containerizingMode>
        <extraDirectories>
            <paths>
                <path>${project.basedir}/src/main/jib</path>
                <path>${project.build.directory}/jib</path>
                <path>
                    <from>target/dependency</from>
                    <into>/opt/amq/lib/optional</into>
                    <includes>log4j-${version.log4j}.jar</includes>
                </path>
                <path>
                    <from>target/dependency</from>
                    <into>/opt/amq</into>
                    <includes>activemq-all-${version.activemq-all}.jar</includes>
                </path>
            </paths>
            <permissions>
                <permission>
                    <!-- don't forget to restrict writing to prevent tampering -->
                    <file>/opt/amq/conf/log4j.properties</file>
                    <mode>444</mode>
                </permission>
                <!-- the copied jars need to be executable -->
                <permission>
                    <file>/opt/amq/lib/${application.executable}</file>
                    <mode>755</mode>
                </permission>
                <permission>
                    <file>/opt/amq/activemq-all-${version.activemq-all}.jar</file>
                    <mode>755</mode>
                </permission>
                <permission>
                    <file>/opt/amq/lib/optional/log4j-${version.log4j}.jar</file>
                    <mode>755</mode>
                </permission>
            </permissions>
        </extraDirectories>
    </configuration>
    <executions>
        <execution>
            <id>jib-build</id>
            <phase>package</phase>
            <goals>
                <goal>${jib.goal}</goal>
            </goals>
        </execution>
    </executions>
</plugin>