tomcat classloader performance issue

Question

Our benchmarking uncovered a bootleneck in Apache Tomcat 7.0.59. When the server reaches its performance limit, most of its threads are locked by ClassLoader.

Stack traces of blocked threads looks like this example:

"http-bio-4504-exec-500" Id=2335 BLOCKED on java.util.jar.JarFile@464f9f8 owned by "[1432628598653] POST /services/signin HTTP/1.0" Id=1990
    at java.util.zip.ZipFile.getEntry(ZipFile.java:304)
    -  blocked on java.util.jar.JarFile@464f9f8
    at java.util.jar.JarFile.getEntry(JarFile.java:226)
    at java.util.jar.JarFile.getJarEntry(JarFile.java:209)
    at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:840)
    at sun.misc.URLClassPath$JarLoader.findResource(URLClassPath.java:818)
    at sun.misc.URLClassPath$1.next(URLClassPath.java:226)
    at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:236)
    at java.net.URLClassLoader$3$1.run(URLClassLoader.java:583)
    at java.net.URLClassLoader$3$1.run(URLClassLoader.java:581)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader$3.next(URLClassLoader.java:580)
    at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:605)
    at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
    at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
    at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
    at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
    at com.sun.xml.ws.util.ServiceFinder$LazyIterator.hasNext(ServiceFinder.java:443)
    at com.sun.xml.ws.util.ServiceFinder$CompositeIterator.hasNext(ServiceFinder.java:390)
    at com.sun.xml.ws.api.message.saaj.SAAJFactory.getMessageFactory(SAAJFactory.java:96)
    at com.sun.xml.ws.api.SOAPVersion.getMessageFactory(SOAPVersion.java:221)
    at com.sun.xml.ws.api.message.saaj.SAAJFactory.readAsSOAPMessage(SAAJFactory.java:275)
    at com.sun.xml.ws.api.message.saaj.SAAJFactory.readAsSAAJ(SAAJFactory.java:205)
    at com.sun.xml.ws.api.message.saaj.SAAJFactory.read(SAAJFactory.java:194)
    at com.sun.xml.ws.message.AbstractMessageImpl.toSAAJ(AbstractMessageImpl.java:199)
    at com.sun.xml.ws.api.message.MessageWrapper.readAsSOAPMessage(MessageWrapper.java:160)
    at com.sun.xml.ws.handler.SOAPMessageContextImpl.getMessage(SOAPMessageContextImpl.java:86)

and the blocker thread is running at the same place

"[1432628598653] POST /services/signin HTTP/1.0" Id=1990 RUNNABLE
    at java.util.zip.ZipFile.getEntry(ZipFile.java:304)
    -  locked java.util.jar.JarFile@464f9f8
    at java.util.jar.JarFile.getEntry(JarFile.java:226)
    at java.util.jar.JarFile.getJarEntry(JarFile.java:209)
    at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:840)
    at sun.misc.URLClassPath$JarLoader.findResource(URLClassPath.java:818)
    at sun.misc.URLClassPath$1.next(URLClassPath.java:226)
    at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:236)
    at java.net.URLClassLoader$3$1.run(URLClassLoader.java:583)
    at java.net.URLClassLoader$3$1.run(URLClassLoader.java:581)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader$3.next(URLClassLoader.java:580)
    at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:605)
    at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
    at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
    at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
    at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
    at com.sun.xml.ws.util.ServiceFinder$LazyIterator.hasNext(ServiceFinder.java:443)
    at com.sun.xml.ws.util.ServiceFinder$CompositeIterator.hasNext(ServiceFinder.java:390)
    at com.sun.xml.ws.api.message.saaj.SAAJFactory.read(SAAJFactory.java:190)
    at com.sun.xml.ws.message.AbstractMessageImpl.toSAAJ(AbstractMessageImpl.java:199)
    at com.sun.xml.ws.api.message.MessageWrapper.readAsSOAPMessage(MessageWrapper.java:160)
    at com.sun.xml.ws.handler.SOAPMessageContextImpl.getMessage(SOAPMessageContextImpl.java:86)

Do our applications do something in a wrong way? Is there a workaround to avoid blocking in classloaders?

Answer 1

The problem was fixed completely with the following hack:

1. I added -Djaxp.debug=true to JVM args.
2. Then I found in logs all classes that JAX-WS was looking for.
3. I added system property for every mentioned class, and it made ServiceFinder to omit searching for service.properties files in classpath.

My final JVM args looked like:

-Djavax.xml.soap.MetaFactory=com.sun.xml.internal.messaging.saaj.soap.SAAJMetaFactoryImpl
-Djavax.xml.datatype.DatatypeFactory=com.sun.org.apache.xerces.internal.jaxp.datatype.DatatypeFactoryImpl
-Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl
-Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl
-Djavax.xml.stream.XMLInputFactory=com.ctc.wstx.stax.WstxInputFactory
-Djavax.xml.stream.XMLOutputFactory=com.ctc.wstx.stax.WstxOutputFactory
-Djavax.xml.transform.TransformerFactory=com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl