I have a linux (ubuntu) server running tomcat7. The server has 1GB of ram, and I am getting heap space errors thrown from java.
Basically what is happening, is that my tomcat server (running my servlet code) is throwing exceptions because there is no more memory to allocate. My code is pretty lean, so I don't think that is the problem.
When I look at htop, I see a lot of java logging processes, I see about 30 of these processes:
ps ax | grep java
7412 pts/0 Sl 0:02 /usr/bin/java -Djava.util.logging.config.file=/usr/share/tomcat/apache-tomcat-7.0.29/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/usr/share/tomcat/apache-tomcat-7.0.29/endorsed -classpath /usr/share/tomcat/apache-tomcat-7.0.29/bin/bootstrap.jar:/usr/share/tomcat/apache-tomcat-7.0.29/bin/tomcat-juli.jar -Dcatalina.base=/usr/share/tomcat/apache-tomcat-7.0.29 -Dcatalina.home=/usr/share/tomcat/apache-tomcat-7.0.29 -Djava.io.tmpdir=/usr/share/tomcat/apache-tomcat-7.0.29/temp org.apache.catalina.startup.Bootstrap start
7426 pts/0 D+ 0:00 grep --color=auto java
25937 ? Sl 13:12 /usr/bin/java -Djava.util.logging.config.file=/usr/share/tomcat/apache-tomcat-7.0.29/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/usr/share/tomcat/apache-tomcat-7.0.29/endorsed -classpath /usr/share/tomcat/apache-tomcat-7.0.29/bin/bootstrap.jar:/usr/share/tomcat/apache-tomcat-7.0.29/bin/tomcat-juli.jar -Dcatalina.base=/usr/share/tomcat/apache-tomcat-7.0.29 -Dcatalina.home=/usr/share/tomcat/apache-tomcat-7.0.29 -Djava.io.tmpdir=/usr/share/tomcat/apache-tomcat-7.0.29/temp org.apache.catalina.startup.Bootstrap start
These java processes slowly leaks memory over time, and because I only have 1gb of ram to play with my tomcat server starts throwing heap space exceptions.
I have fiddled with log4j (but it should be turned off) and I have done all I can to turn off logging in configs I can find.
Any help would be great, I just need to get rid of these processes - they are way to greedy. Thanks!
~ Dan
EDIT: MORE INFO:
This is probably a big enough hint to solve this one entirely - I checked some of the startup debug and came up with this:
Oct 02, 2012 8:50:08 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/NMVServer] appears to have started a thread named [http-bio-8080-exec-53] but has failed to stop it. This is very likely to create a memory leak.
Oct 02, 2012 8:50:08 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/NMVServer] appears to have started a thread named [http-bio-8080-exec-54] but has failed to stop it. This is very likely to create a memory leak.
Oct 02, 2012 8:50:08 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/NMVServer] appears to have started a thread named [http-bio-8080-exec-62] but has failed to stop it. This is very likely to create a memory leak.
Oct 02, 2012 8:50:08 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/NMVServer] appears to have started a thread named [http-bio-8080-exec-64] but has failed to stop it. This is very likely to create a memory leak.
One problem with leaks that I found is due to a bug with the JDBC Driver http://bugs.mysql.com/bug.php?id=36565 according to a couple of forums. That solves another message I was getting.
I have been looking around and I have found this which is quite useful:
http://wiki.apache.org/tomcat/MemoryLeakProtection
Now I am just trying to work through all your advise thanks heaps!
~ Dan
I general I would expect the logging code in java to have been well tested. If there is no log output being generated by your logging config it has a low overhead. Your htop results are probably showing some idle thread status in tomcat.
It could be logging but you may want to consider all the ways your application or another component may be leaking memory.
All JVM's have pretty good tools for diagnosing memory leaks. Threads like this discuss great ways to find out what is going on.
Edit: some more thoughts
You might want to check your configuration for too many sessions if a lot of requests are using HttpServletRequest.getSession(). The Manager component config goes inside Context and defaults to -1 for unlimited sessions.
Check your version of tomcat with the current issue fix list in case you are encountering a known memory leak.
Lastly, Tomcat 7 includes a JreMemoryLeakPreventionListener that might show resources not being released.