Identifying memory leaks using Eclipse MAT - Part 2

As I wrote in Part 1, we identified that there was some sort of memory leak happening within our Java JEE application – so the next steps were to obtain a Heap Dump and run it through Eclipse MAT.

So what is a Heap Dump? From the Eclipse docs

A heap dump is a snapshot of the memory of a Java process at a certain point of time. There are different formats for persisting this data, and depending on the format it may contain different pieces of information, but in general the snapshot contains information about the java objects and classes in the heap at the moment the snapshot was triggered.

The Memory Analyzer is able to work with HPROF binary heap dumps, IBM system dumps (after preprocessing them), and IBM portable heap dumps (PHD) from a variety of platforms.

Typical information which can be found in heap dumps (once more - depending on the heap dump type) is:

• All Objects

Class, fields, primitive values and references

• All Classes

Classloader, name, super class, static fields

• Garbage Collection Roots

Objects defined to be reachable by the JVM

• Thread Stacks and Local Variables

How do we get a heap dump into a .hprof file?

Set the following flags to the java process

·        -XX:+HeapDumpOnOutOfMemoryError writes heap dump on OutOfMemoryError

·        -XX:+HeapDumpOnCtrlBreak writes heap dump together with thread dump on CTRL+BREAK

Or you can fire one via tools jmap or jconsole

·        Sun JMap: <jdkhome>/bin/jmap  -dump:format=b,file=HeapDumpFilename.hprof

 

Our idea was to take multiple heap dumps every 2 hours or so and observe the objects in the heap. As with a thread dump, a heap dump is a static view of the live objects in the heap at that time – so it’s not possible to make a definitive judgment on just viewing one heap dump – hence space it out over some time and observe if the same objects (sometimes these have the same memory address location) are growing and causing a possible leak.

Also remember – at the point of taking a heap dump, Java runs a full GC, so you are left with the live objects in the heap.

We took the first heap dumps every 2 hours, but nothing much was happening for the first 12 hours or so.

Once you have the hprof with you, and have installed Eclipse Memory Analyzer Tool (MAT) just open the hprof in Eclipse and wait for it to do it’s thing.

The page will open with an Overview.

This lists important stuff like the Size of the heap – so in our case, though we have a 2 GB heap the retained size after GC is 473.1 MB.

The graph of “Biggest Objects by Retained Size” will give you immediate clues and this is also reflected in tabular format in the “Dominator Tree” report.



Overview

But the best thing about MAT is the Leak Suspects report which points out clearly potential memory leaks.

 

 

 

So in our hprof, 55% of the heap was being retained by one instance of weblogic.xml.query.xdbc.Context i.e. if this object was reclaimed by GC, we could get back 273 MB of memory. This object was residing at memory location weblogic.xml.query.xdbc.Context @ 0xb2477930

The important thing to see in this view is the Retained Heap

In Memory Analyzer the term shallow size means the size of an object itself, without counting and accumulating the size of other objects referenced from it.

The retained heap is the total objects and memory which this object is holding onto.

 

 

We took the next snapshot after 2 hours of the first heap dump and it showed the same instance of weblogic.xml.query.Context@0x7c1642f8 had now grown to 301.6 MB with the total heap now at 524.7 MB

 

Our last snapshot in confirming our problem was after a further 2 hours, so a total of 4 hours since the first heap dump showed this object.

The heap had now grown to 616.5 MB with 54% of it occupied by the same object

 

 

So next we now found out from MAT what was causing the leak. We next had to analyze how this class weblogic.xml.query.xdbc.Context  was being used in the app and how we could prevent the leak. I will post that in Part 3.