Issue: Getting out of native memory exception and was wondering if excessive garbage collection can lead to this? Also any advice on GC policy or tuning would be helpful. I'm not sure if what I have warrants a change yet.
Good Reference StackOverflow Question: Which GC Policy to Use
Specs:
Preliminary Analysis:
Attached Screenshot:
1.1 - 4 hour period of heap utilization . Each little green garbage can represents a major garbage collection point. 1.2 GC time spend in the above graph. 1.3 What the heap utilization looked like during a out of memory exception.
Exception:
EJB threw an unexpected (non-declared) ejb Exception data: java.lang.OutOfMemoryError: native memory exhausted at garbagecollection.mycode.test at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1658) at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1598) at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:149) at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:190) at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:125) at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:190) at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:125) at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:190) at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:125) at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:80) at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:908) at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:935) at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:503) at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:181) at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91) at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:875) at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592) at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:453) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:515) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:306) at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:83) at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1784) at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165) at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217) at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161) at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138) at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204) at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775) at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1646)
The error message you get indicate a problem with native memory, that is, memory outside the heap. The garbage collector is not responsible for off-the-heap memory, why you can not affect this error with garbage collector settings. Excessive garbage collecting should not lead to native memory issues (unless there is a bug in the GC, of course).
What is causing this particular error is hard to say without more information. I assume you have at least 12 GB of available RAM on the machine? Otherwise you are simply running a heap that is too big for the available memory.
Some examples of things in Java that can exhaust native memory:
Leaks in native (JNI) code
Direct allocation of many ByteBuffer
s
Or maybe there is no leak, this behavior is expected for your application, and you only need to allow larger processes or buy some more memory.
First step to troublehsoot would be to check the process size against possible process size limits. You could simply run into such a limit without there being an actual leak.
If that is not the problem, track the process size over time, especially in relation to available memory to see if you have a "leaky" behavior.
If you actually think you have a native memory leak, identify which libraries that use JNI you are using (e.g. JDBC drivers) and try to replace them with Java-only versions. Also, check for known memory leak bugs in such libraries and your JVM version and WebSphere version.
If that fails you will have to resort to native memory leak troubleshooting tools. That is another question.