How to debug permgen memory errors for a project that uses sbt and play 2 framework in Scala

Question

I'm using sbt with enough memory and PermSize (see values below) but I still keep getting memory errors.

[error] java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: PermGen space

My sbt script looks like the following.

SBT_OPTS="-Xms1024M -Xmx1536M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=4028M" java $SBT_OPTS -jar `dirname $0`/sbt-launch.jar "$@"

I'm making changes in the IntelliJ 13 IDE and then using the following commands to run my Play App

$ ./sbt
[name-of-my-play-project] $ compile
[name-of-my-play-project] $ ~run

I've also tried just run instead of ~run. I can run the project and visit pages if I do a cold start using the $./sbt followed by ~run. However, after a few code changes Play doesn't respond anymore.

I tried running the same project on another machine (also OS-X) where I get similar behavior. My concern is that it doesn't happen all the time, but happens often enough that it makes me unproductive because I've to restart sbt and play from scratch which is totally defeats one of the key benefits of the Play2 framework.

I would appreciate if someone can help with where should I start looking to identify what's wrong with my setup.

I created the project using activator and then started using sbt for it.

UPDATE - Screenshot from YourKit (based on suggestion from @James) that shows the memory leak.

UPDATE 2 YourKit shows a deadlock (please see an attached screenshot). I'm not sure if this is the root cause of the issue. I'm putting it here just in case someone can provide more insights into what's going on.

UPDATE 3 - SOLUTION After more than 12 hours of non-stop debugging I was finally able to figure it out. It's was a very simple thing that I'll explain in the end. Since I had a lot of 3rd party libraries /plugins (ReactiveMongo, Redis, ...) in my code and I was not sure if any of that was causing the OutOfMemory error. So I just removed all the unwanted code till I was left with a simple controller with one method that just returns a simple "Hello world". I still got the same error with this simple program. This meant that there was something wrong with my configuration. I tried changing the sbt version to 0.13.5 (sbt.version=0.13.5) in my project/build.properties and that still didn't fix it. I downloaded the new sbt-launch.jar (0.13.5) and the problem still existed. Again, one thing that was constant through out was the error never changed. So for some reason the flags for sbt where not being set.

My sbt bash script looked like (EVERYTHING IN ONE LINE) -

SBT_OPTS="-Xms1024M -Xmx1536M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=4028M" java $SBT_OPTS -jar `dirname $0`/sbt-launch.jar "$@"

I changed it to

SBT_OPTS="-Xms1024M -Xmx1536M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=4028M" 
java $SBT_OPTS -jar `dirname $0`/sbt-launch.jar "$@"

This fixed it. Again it was class loading issue but not in my code but in the configuration of sbt.

Answer 1

What you're seeing is a class loader leak. There are many possible causes of class loader leaks. The most common is starting up threads and not shutting them down. Also thread locals, caches outside the class loader, and shutdown hooks are common culprits.

Attach a profiler (I recommend YourKit) and watch what happens to the threads after each reload. If new ones are created but not shutdown, then find out which threads they are, what is starting them, and make sure that gets cleaned up by Global.onStop.

If that doesn't help, take a heap dump, and load it in YourKit. Now look for multiple instances of classes that you expect to only ever be one of, or that there is an excessive number of. Find the merged GC roots to that class, and somewhere along that chain should be the source of the memory leak.