Unexplainable ClassNotFoundException in server unmanaged thread

Question

I am facing a weird ClassNotFoundException: org.apache.commons.io.IOUtils under the following circumstances:

I have a web application running in Websphere Application Server 8.0 that uses commons-io 2.0.1 (this jar is correctly placed in the classpath).

I am using org.apache.commons.io.input.Tailer to tail a log file and show its contents on screen. Tailer implements Runnable so it runs in its own Thread.

On application shutdown I invoke tailer.stop() to stop the Thread. Then the last thing that Tailer executes is a finally block in which it closes the log reader.

} finally {
    IOUtils.closeQuietly(reader);
}

When it executes that line there's a ClassNotFoundException: org.apache.commons.io.IOUtils.

I've made some tests adding some code into Tailer class (and replacing it in the jar) to determine if it's a ClassLoader issue:

1. Determine Tailer's resource:

Tailer.class.getClassLoader().getResource("org/apache/commons/io/input/Tailer.class");

Output: wsjar:file:/home/myuser/.m2/repository/commons-io/commons-io/2.0.1/commons-io-2.0.1.jar!/org/apache/commons/io/input/Tailer.class which is the right jar

2. What classloader is it using?: Tailer.class.getClassLoader();

com.ibm.ws.classloader.CompoundClassLoader@8d336acd[war:application/myApp.war] Again this is correct.

3. Try to load IOUtils directly from the previous ClassLoader: Tailer.class.getClassLoader().loadClass("org.apache.commons.io.IOUtils");

Again a ClassNotFoundException

4. Now, the most surprising thing: if I add a Class.forName() like below at the beginning of the run method, the class is correctly loaded and now the IOUtils call in the finally block works. How come??

public void run() {
     RandomAccessFile reader = null;
     try {
         Class.forName("org.apache.commons.io.IOUtils");
     }catch(Exception e){
         e.printStackTrace();
     }         
     try {
         ...
         while (run) {
            ...
         }
     } catch (Exception e) {
         listener.handle(e);
     } finally {
         IOUtils.closeQuietly(reader);
     }
 }

 /**
 * Allows the tailer to complete its current loop and return.
 */
 public void stop() {
     this.run = false;
 }

It doesn't make any sense! It is using Tailer from commons-io-2.0.1.jar but it isn't it able to use a class that it's inside the very same jar. But if you force a load of the class at the beginning of the run(), then it doesn't fail anymore. Any ideas?

This is the whole stacktrace:

Exception in thread "Thread-88" java.lang.NoClassDefFoundError: org.apache.commons.io.IOUtils
    at org.apache.commons.io.input.Tailer.run(Tailer.java:319)
    at java.lang.Thread.run(Thread.java:784)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.io.IOUtils
    at java.net.URLClassLoader.findClass(URLClassLoader.java:434)
    at com.ibm.ws.bootstrap.ExtClassLoader.findClass(ExtClassLoader.java:230)
    at java.lang.ClassLoader.loadClassHelper(ClassLoader.java:703)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:682)
    at com.ibm.ws.bootstrap.ExtClassLoader.loadClass(ExtClassLoader.java:123)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:665)
    at com.ibm.ws.classloader.ProtectionClassLoader.loadClass(ProtectionClassLoader.java:62)
    at com.ibm.ws.classloader.ProtectionClassLoader.loadClass(ProtectionClassLoader.java:58)
    at com.ibm.ws.classloader.CompoundClassLoader.loadClass(CompoundClassLoader.java:566)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:665)
    at com.ibm.ws.classloader.CompoundClassLoader.loadClass(CompoundClassLoader.java:566)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:665)
    ... 2 more

Answer 1

Items #1 and #2 are simply data points that show that common problems are not the cause. They are not necessarily indicative of the source of the issue.

Item #3 is indicative of the issue. It would seem that based on that item, the classes have already been unloaded from memory at that point. I looked through for some documentation that would show under what conditions ClassLoader would throw an exception, but was unable to find any. So, I am guessing that classes have been unloaded and the state of the application is such that the warfile itself has been freed. Further guesses are irrelevant.

Item #4 is indeed curious. The documentation on Class.forName says "Instances of the class Class represent classes and interfaces in a running Java application." and "Class has no public constructor. Instead Class objects are constructed automatically by the Java Virtual Machine as classes are loaded and by calls to the defineClass method in the class loader.". Which seems to mean that even if the Class has been unloaded from memory Class.forName should be able to load it.

Item #4 in your example, does not have Class.forName in the finally block. I assume that is simply an artifact of the pasted code and that you did in fact place it there and it threw an error.

Update : After discussing with the OP, The following suggestion does the job. At the point of invocation of tailer.stop(), place a thread.sleep() for a few seconds after the call. This should allow enough time for the finally call to finish up (closeQuietly(reader)) before the Application has been unloaded from the ClassLoader.