This is related to question Java Appengine APPSTATS causing java out of memory error.
Appstats seems to cause a java.lang.OutOfMemoryError on 128MB instances and I wonder whether there are ways to reduce the amount of logging. Is there a way to filter some package names from the stack trace?
GAE API:
@14ms memcache.Get real=7ms api=0ms
Stack:
com.google.appengine.tools.appstats.Recorder:290 makeAsyncCall()
com.google.apphosting.api.ApiProxy:184 makeAsyncCall()
com.google.apphosting.api.ApiProxy:123 makeAsyncCall()
com.google.appengine.api.memcache.MemcacheServiceApiHelper:104 makeAsyncCall()
com.google.appengine.api.memcache.AsyncMemcacheServiceImpl:372 doGetAll()
com.google.appengine.api.memcache.AsyncMemcacheServiceImpl:333 getIdentifiables()
com.google.appengine.api.memcache.MemcacheServiceImpl:61 getIdentifiables()
my appplication API:
com.googlecode.objectify.cache.EntityMemcache:215 getAll()
com.googlecode.objectify.cache.CachingAsyncDatastoreService:253 get()
com.googlecode.objectify.cache.CachingDatastoreService:161 get()
com.googlecode.objectify.cache.CachingDatastoreService:147 get()
com.googlecode.objectify.cache.CachingDatastoreService:128 get()
siena.gae.GaePersistenceManager:231 getByKey()
siena.Model:106 getByKey()
com.sirtrack.iridium.model.GroupEntity:147 getByKey()
com.sirtrack.iridium.task.ProjectUpdateTask:49 doPost()
irrelevant:
javax.servlet.http.HttpServlet:637 service()
javax.servlet.http.HttpServlet:717 service()
org.mortbay.jetty.servlet.ServletHolder:511 handle()
org.mortbay.jetty.servlet.ServletHandler$CachedChain:1166 doFilter()
com.google.appengine.tools.appstats.AppstatsFilter:141 doFilter()
org.mortbay.jetty.servlet.ServletHandler$CachedChain:1157 doFilter()
org.vosao.filter.SiteFilter:106 doFilter()
org.mortbay.jetty.servlet.ServletHandler$CachedChain:1157 doFilter()
org.vosao.filter.RewriteFilter:79 doFilter()
org.mortbay.jetty.servlet.ServletHandler$CachedChain:1157 doFilter()
org.vosao.filter.AuthenticationFilter:83 doFilter()
org.mortbay.jetty.servlet.ServletHandler$CachedChain:1157 doFilter()
org.vosao.filter.PluginCronFilter:78 doFilter()
org.mortbay.jetty.servlet.ServletHandler$CachedChain:1157 doFilter()
org.vosao.filter.LanguageFilter:66 doFilter()
org.mortbay.jetty.servlet.ServletHandler$CachedChain:1157 doFilter()
org.vosao.filter.UpdateFilter:78 doFilter()
org.mortbay.jetty.servlet.ServletHandler$CachedChain:1157 doFilter()
org.vosao.filter.InitFilter:80 doFilter()
org.mortbay.jetty.servlet.ServletHandler$CachedChain:1157 doFilter()
org.vosao.filter.ContextFilter:74 doFilter()
org.mortbay.jetty.servlet.ServletHandler$CachedChain:1157 doFilter()
com.google.apphosting.utils.servlet.ParseBlobUploadFilter:102 doFilter()
org.mortbay.jetty.servlet.ServletHandler$CachedChain:1157 doFilter()
com.google.apphosting.runtime.jetty.SaveSessionFilter:35 doFilter()
org.mortbay.jetty.servlet.ServletHandler$CachedChain:1157 doFilter()
com.google.apphosting.utils.servlet.TransactionCleanupFilter:43 doFilter()
org.mortbay.jetty.servlet.ServletHandler$CachedChain:1157 doFilter()
org.mortbay.jetty.servlet.ServletHandler:388 handle()
org.mortbay.jetty.security.SecurityHandler:216 handle()
org.mortbay.jetty.servlet.SessionHandler:182 handle()
org.mortbay.jetty.handler.ContextHandler:765 handle()
org.mortbay.jetty.webapp.WebAppContext:418 handle()
com.google.apphosting.runtime.jetty.AppVersionHandlerMap:249 handle()
org.mortbay.jetty.handler.HandlerWrapper:152 handle()
org.mortbay.jetty.Server:326 handle()
org.mortbay.jetty.HttpConnection:542 handleRequest()
org.mortbay.jetty.HttpConnection$RequestHandler:923 headerComplete()
com.google.apphosting.runtime.jetty.RpcRequestParser:76 parseAvailable()
org.mortbay.jetty.HttpConnection:404 handle()
com.google.apphosting.runtime.jetty.JettyServletEngineAdapter:135 serviceRequest()
com.google.apphosting.runtime.JavaRuntime$RequestRunnable:446 run()
com.google.tracing.TraceContext$TraceContextRunnable:449 runInContext()
com.google.tracing.TraceContext$TraceContextRunnable$1:455 run()
com.google.tracing.TraceContext:695 runInContext()
com.google.tracing.TraceContext$AbstractTraceContextCallback:333 runInInheritedContextNoUnref()
com.google.tracing.TraceContext$AbstractTraceContextCallback:325 runInInheritedContext()
com.google.tracing.TraceContext$TraceContextRunnable:453 run()
com.google.apphosting.runtime.ThreadGroupPool$PoolEntry:251 run()
java.lang.Thread:679 run()
Here's a stack trace of the maybe-caused-by-stacktrace OutOfMemoryError:
Error for /_ah/queue/projectupdate
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3057)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:117)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:407)
at java.lang.StringBuffer.append(StringBuffer.java:241)
at java.io.StringWriter.write(StringWriter.java:112)
at java.io.PrintWriter.write(PrintWriter.java:429)
at java.io.PrintWriter.write(PrintWriter.java:446)
at java.io.PrintWriter.print(PrintWriter.java:576)
at java.io.PrintWriter.println(PrintWriter.java:712)
at java.lang.Throwable.printStackTrace(Throwable.java:529)
at com.google.appengine.tools.appstats.Recorder.createStackTrace(Recorder.java:160)
at com.google.appengine.tools.appstats.Recorder.initializeIntermediary(Recorder.java:271)
at com.google.appengine.tools.appstats.Recorder.makeAsyncCall(Recorder.java:290)
at com.googlecode.objectify.cache.TriggerFutureHook.makeAsyncCall(TriggerFutureHook.java:144)
at com.google.apphosting.api.ApiProxy.makeAsyncCall(ApiProxy.java:184)
For Google Engineers, here's a link to those errors
I've found an undocumented (AFAIK) parameter called "maxLinesOfStackTrace" in the class com.google.appengine.tools.appstats.Recorder.java.
In web.xml:
<filter>
<filter-name>appstats</filter-name>
<filter-class>com.google.appengine.tools.appstats.AppstatsFilter</filter-class>
<init-param>
<param-name>maxLinesOfStackTrace</param-name>
<param-value>16</param-value>
</init-param>
</filter>
It seems to work locally, I'll let you know whether it also helps avoiding the OutOfMemoryError.