Search code examples
javamultithreadinggarbage-collectionweblogic12c

java heap and thread analysis for memory leak

My WebLogic server was configured with 16gb of heap space, but it was 90% used within 1 hour of production usage when most of the users started work. I observed there were several stuck threads whenever this happens.

I have captured the heap dump when the heap was approx 10% free. How do I inspect the heap dump to find out the memory leak, or process, codes which is causing this issue.

I have tried to understand the memory leak, running tools like JMap and Eclipse MAT, but it maybe due to lack of experience, I couldn't understand what these tools are trying to show. Or how/what should I look out for?

I have both the before/after GC heap dump to analyze.

I have reviewed the thread dumps, there were no "waiting to lock" objects threads, the threads were similar as shown below, with threads stuck with no obvious reasons.

Solution

  • According to your heap dump, your biggest memory issue is the int arrays, indeed it takes nearly 70 % of your heap (Yes sort the Size Column instead).

    1. Select it in your heap dump, right click and select on Show in Instances View
    2. Then browse the biggest objects and for each of them right click and select Show Nearest GC Root to see which Object has still an hard reference to the int array which prevents to be eligible for the GC.

    It could help you to find your memory leak assuming that it is a memory leak.

    See below an example of Nearest GC Root allowing to identify a leak that I added intentionally to my program just to show the idea. As you can see in the screenshot, I have an array of int which cannot be eligible for the GC because it is stored in an HashMap called leak in my class Application, so I know that my memory issue could be due to this particular HashMap especially if I have many other objects which lead to this HashMap.

    enter image description here

    NB: Be patient when you try to identify a leak as it is not always obvious, the ideal situation is where you have a huge object that takes the whole heap but obviously it is not your case there is nothing really obvious that is the reason why I propose to investigate the int arrays first. Don't forget that it could also be little int arrays but thousands of them with the same Nearest GC Root.

    Another trick, If you have JProfiler you can simply follow this wonderful tutorial to find your leak.

    Response Update:

    One simple way to better identify the root cause of the memory leak is to take at least 2 heap dumps then compare them using a tool like jhat with the syntax 

    jhat -J-Xmx2G -baseline ${path-to-the-first-heap-dump} ${path-to-the-second-heap-dump}

    It will launch a small HTTP sever on port 7000 so:

    1. Launch http://localhost:7000/
    2. Then click on Show instance counts for all classes (including platform)

    You will then see the list of Classes ordered by total amount of new instances created. You can then use VisualVM to do what I described in the first part of my answer to find the root cause of your memory leak.

    You can also use jhat

    1. By selecting of the Top Classes then for each of them
    2. click on one "Reference to this Object"
    3. then click on Exclude weak refs

    You will then see the GC root of each instances like the next screenshot:

    enter image description here

    Another way is to use Eclipse Memory Analyzer also called MAT.

    1. Open the second snapshot with it
    2. Select the view histogram
    3. Then for each of the Top Classes right click
    4. Choose Merge Shortest Paths To GC Roots/ Exclude All references

    you will then see something like the next screenshot:

    enter image description here