Search code examples
javagarbage-collectionjettypool

Jetty server hangs with large object pool without any requests being made

I am running a web application using jetty server with maven. The application contains a large pool of static objects in Lists and Maps contributing to the 2.8GB of physical memory usage. After several hours, the server hangs with maximum CPU usage. This happens without any user interaction or requests made on the server.

I have noticed that during those several hours, while the server is running fine, the memory slowly reduces to 1.7GB. My suspicion is that this could be a garbage collection related issue.

Questions:

  1. Could it be that GC hangs while erroneously collecting or inspecting my large object pools and its references?
  2. How would I go about debugging and fixing this issue?

Note that on Windows I do not have this problem. Once the application starts and fills up its pool, it occupies 3.4GB and stays exactly the same without ever crashing.

Server startup and environment:

setenforce 0
export MAVEN_OPTS="-Xmx5120m -Xms5120m -XX:+UseConcMarkSweepGC -Xgcthreads1 -XX:MaxGCPauseMillis=2000 -XX:GCTimeRatio=10"
sudo nohup mvn -D jetty.port=80 jetty:run &

Operating system:

Ubuntu 12.04.1 LTS

Java:

OpenJDK Runtime Environment (IcedTea6 1.11.5) (6b24-1.11.5-0ubuntu1~12.04.1)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

Maven: 

Apache Maven 3.0.4

Jetty:

8.1.8.v20121106
Solution

  • It's hard to say if it's due to incorrect GC that cause the system hang. I think you can make some moves to get more information:

    1. Add -verbose:gc -Xloggc:/home/admin/logs/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps to your JVM arguments, those will help you find more information of GC.
    2. Collecting thread dump regularly to see what's going on in your application at runtime.
    3. Get a memory dump when the machine is about to die, which can be analyzed by MAT.
    4. When CPU spikes, use top -H -p<pid> to find the dominator threads and spot them in the thread dump, then you can basically find out which line of the code is doing wrong.

    Here is a really good article of How to Analyze Java Thread Dumps.