We have a very weird problem here.
jvm : tried 1.7.0_55-b13 and 1.7.0_75-b13
tomcat : 7.0.56
os : Ubuntu 12.04.5 LTS (64bit) (3.5.0-23-generic)
Using tomcat with a big application built with spring.
On one of our production environment, we sometime get "NoClassDefFound" always with the same stack trace.
It only happen after some time, and testable with a specific workflow. However, the class marked as "not found" is there (in a jar in WEB-INF/lib) and as been used multiple times before the problem appear and the exception start to be thrown : the specific workflow mentionned above can be executed successfully many times during the day. Somehow the workflow stop working and start throwing NoClassDefFound exceptions.
It seems that the class is loaded, used, and then, overtime, disappear from the jvm.
The jvm runs with the following arguments :
-XX:+UseConcMarkSweepGC
-Xmx6900m
-Xms2000m
-XX:MaxPermSize=900m
-XX:+UseParNewGC
-XX:+CMSParallelRemarkEnabled
-XX:NewRatio=1
-XX:TargetSurvivorRatio=75
-XX:SurvivorRatio=8
-XX:+AggressiveOpts
-XX:ReservedCodeCacheSize=256m
-Djava.util.logging.config.file=/var/lib/tomcat7/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djava.awt.headless=true
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.port=2037
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname=10.71.1.112
-Dcom.sun.management.jmxremote.password.file=/etc/tomcat7/jmxremote.password
-Dcom.sun.management.jmxremote.access.file=/etc/tomcat7/jmxremote.access
-Djava.net.preferIPv4Stack=true
-Djava.endorsed.dirs=/usr/share/tomcat7/endorsed
-classpath /usr/share/tomcat7/bin/bootstrap.jar:/usr/share/tomcat7/bin/tomcat-juli.jar
-Dcatalina.base=/var/lib/tomcat7 -Dcatalina.home=/usr/share/tomcat7
-Djava.io.tmpdir=/tmp/tomcat7-tomcat7-tmp org.apache.catalina.startup.Bootstrap start
Anyone is aware of similar problems ?
I corrected the problem and here I post the solution. Maybe it will help someone somehow somewhere sometime :)
The problem is not what we tought at first : Class definition did not disappear from the class loader, but were not initialized at first (startup).
Here is the workflow that caused the error : When tomcat last stopped, it serialized its sessions to a file (as per the default session storage mecanism).
When the application was started again, tomcat failed to reload its sessions because of an "ExceptionInInitializerError" when initializing a class for deserializing its instances.
Then, all subsequent calls to that class (that were not properly initialized) produced the stack trace we see in the question (NoClassDefFound).
The reason of the ExceptionInInitializerError was that the serialized class tried to call a spring context statically (context that was not initialised at the time, then produced a NullPointerException that caused the ExceptionInInitializer).