My java program mysteriously crashes due to a NoClassDefFoundError. The mystery is that the exception message indicates that the problematic class is java/util/concurrent/CopyOnWriteArrayList$COWIterator, which is part of the JRE. The exception is thrown after other JRE classes have been loaded without problems.
The exception is thrown from logback (version 1.1.3), seemingly when it is iterating through the list of appenders to write to (I guess it uses a CopyOnWriteArrayList to hold the list of appenders):
Thread [main] (Suspended (exception NoClassDefFoundError))
CopyOnWriteArrayList<E>.iterator() line: 959
AppenderAttachableImpl<E>.appendLoopOnAppenders(E) line: 47
Logger.appendLoopOnAppenders(ILoggingEvent) line: 273
Logger.callAppenders(ILoggingEvent) line: 260
Logger.buildLoggingEventAndAppend(String, Marker, Level, String, Object[], Throwable) line: 442
Logger.filterAndLog_0_Or3Plus(String, Marker, Level, String, Object[], Throwable) line: 396
Logger.info(String) line: 600
MyProgramTextLogger.logStarting() line: 303
MyProgram.run() line: 1686
MyProgram.runProgram(MyProgram) line: 790
MyProgram.main(String[]) line: 712
Of course, logback code must already have loaded the CopyOnWriteArrayList class when it created the list it is trying to iterate over, so the classloader must have done its job properly at that point.
To further add to the mystery this problem does not occur if I run the program in the foreground, or as a daemon process controlled by systemd. It manifests only when I run my program as a child of a daemon process started by systemd.
Experimentation (by using a static code block) showed that the main thread classloader could load the CopyOnWriteArrayList$COWIterator class OK on program start up (the program then throws a NoClassDefFoundError for a different JRE class). It's as if the classloader had decided to lose the ability to load classes.
I do not pass any special arguments to the java program. I have used a full path-name for the java option and provided the -showversion option to ensure that I am using the same JVM and JRE each time. My command-line is simply:
/usr/bin/java \
-showversion \
-jar /home/myuser/lib/my-app.jar --1 --latest
Java reports its version to be
java version "1.7.0_75"
OpenJDK Runtime Environment (rhel-2.5.4.2.el7_0-x86_64 u75-b13)
OpenJDK 64-Bit Server VM (build 24.75-b04, mixed mode)
My program does not do any explicit class loading, nor do I have a custom class loader: it uses the default class loader. However, bkail has pointed out that it should not matter if I did, because the Java platform classes should be loaded by the boot class loader rather than the application class loader.
At the point where my program fails my code has not created any threads. My code does not include any native code, and thus no "external" threads that could confuse things. I believe that logback creates some threads, however, and of course there are the normal Java daemon threads.
And another mysterious aspect. In between creating the logback logger (using org.slf4j.LoggerFactory.getLogger(MyProgram.class)) and calling the Logger.info(String) method that results in the exception, essentially all my application does is to close the standard input, standard output and standard error output streams, using a call to this method:
private static void closeSystemStreams() {
try {
System.in.close();
} catch (IOException e) {
// Do nothing; should never happen, and there is nothing we can do if
// it does
}
System.out.close();
System.err.close();
}
If I omit that closing code, the program runs OK. Now, I can understand that closing those streams might fail in some circumstances (and some recommend against it), but why should it interfere with the bootstrap class loader?
What do I have to do to ensure that logback or the JVM will not have these kinds of classloading problems?
It seems that some of the Java library code gets confused if the standard streams are closed. Perhaps there is a hard coded check of the value of a file descriptor somewhere. That might or might not be a bug, depending on how generously you interpret the behaviour. It would seem to be safest not to close the standard streams.