I have a production issue that I can't replicate in the development or test environments. Also, since it is in production I can't stop/start Tomcat without a lot of red-tape.
I am observing delays of about 12 seconds (give or take 500ms) in Threads of execution that use the AudioSystem class. The delays occur intermittently and never within a few minutes of each other. Other threads proceed concurrently without a problem. The delays appear to become less frequent with higher load.
I was able to gather stack traces while the delay occurs by using the JConsole threads tab. e.g.
Name: http-8081-Processor27
State: RUNNABLE
Total blocked: 134,991 Total waited: 23,132
Stack trace:
java.util.zip.ZipFile.open(Native Method)
java.util.zip.ZipFile.<init>(Unknown Source)
java.util.jar.JarFile.<init>(Unknown Source)
java.util.jar.JarFile.<init>(Unknown Source)
org.apache.catalina.loader.WebappClassLoader.openJARs(WebappClassLoader.java:1789)
org.apache.catalina.loader.WebappClassLoader.findResources(WebappClassLoader.java:1019)
java.lang.ClassLoader.getResources(Unknown Source)
sun.misc.Service$LazyIterator.hasNext(Unknown Source)
com.sun.media.sound.JSSecurityManager$7.run(Unknown Source)
java.security.AccessController.doPrivileged(Native Method)
com.sun.media.sound.JSSecurityManager.getProviders(Unknown Source)
com.sun.media.sound.JDK13Services.getProviders(Unknown Source)
javax.sound.sampled.AudioSystem.getProviders(Unknown Source)
javax.sound.sampled.AudioSystem.getAudioFileReaders(Unknown Source)
javax.sound.sampled.AudioSystem.getAudioInputStream(Unknown Source)
It looks as though the loading of JAR files by the WebappClassLoader is implicated. I tweaked this classloader in the development environment to log out the JARs it is loading. There are 55 JARs loaded, all from a local disk, e.g.:
Fri Mar 02 17:02:04 EST 2012 About to open: E:\mycompany\Apps\Apache\Tomcat 5.5\webapps\application\WEB-INF\lib\log4j-1.2.15.jar
The server has a misconfigured proxy server. This can reliably produce 12 second delays when fetching webpages. However, DEV and TEST environments are also misconfigured but do not yet show the delays. Would this cause Java to show delays when accessing local JAR files?
Tomcat has been allocated 1500MB maximum memory. The GC is ConcurrentMarkSweep and never uses more than 800MB.
The server is built with Windows 2003, Tomcat 5.5.23, Java 1.5.0-v14. Any clues as to what is happening would be greatly appreciated.
It sounds like the class loader is attempting to talk to the internet via your proxy. I'd check the way that you've configured the classloader. Bearing in mind that the classloader will implicitly delegate up its parent classloader chain first, so you'll need to check them as well.
It might help you track down the problem if you temporarily told Java to use an internal proxy so you could see what the classloader trying to fetch from the internet.