As I wrote
in Part 1, we identified that there was some sort of memory leak happening
within our Java JEE application – so the next steps were to obtain a Heap Dump
and run it through Eclipse MAT.
A heap dump is a snapshot of the memory of a Java
process at a certain point of time. There are different formats for persisting
this data, and depending on the format it may contain different pieces of
information, but in general the snapshot contains information about the java
objects and classes in the heap at the moment the snapshot was triggered.
The Memory Analyzer is able to work with HPROF
binary heap dumps, IBM system dumps (after preprocessing them), and IBM
portable heap dumps (PHD) from a variety of platforms.
Typical information which can be found in heap
dumps (once more - depending on the heap dump type) is:
Class, fields,
primitive values and references
Classloader,
name, super class, static fields
Objects defined
to be reachable by the JVM
- Thread Stacks and Local Variables
How do we get a heap dump into a .hprof file?
Set the
following flags to the java process
·
-XX:+HeapDumpOnOutOfMemoryError writes heap dump on OutOfMemoryError
·
-XX:+HeapDumpOnCtrlBreak writes heap dump together with thread dump on
CTRL+BREAK
Or you can fire one via tools jmap or jconsole
·
Sun JMap: <jdkhome>/bin/jmap -dump:format=b,file=HeapDumpFilename.hprof
Our idea
was to take multiple heap dumps every 2 hours or so and observe the objects in
the heap. As with a thread dump, a heap dump is a static view of the live objects
in the heap at that time – so it’s not possible to make a definitive judgment
on just viewing one heap dump – hence space it out over some time and observe
if the same objects (sometimes these have the same memory address location) are
growing and causing a possible leak.
Also
remember – at the point of taking a heap dump, Java runs
a full GC, so you are left with the live objects in the heap.
We took the first heap dumps every 2 hours,
but nothing much was happening for the first 12 hours or so.
Once you have the hprof with you, and have
installed Eclipse Memory Analyzer Tool (MAT) just open the hprof in Eclipse and
wait for it to do it’s thing.
The page will open with an Overview.
This lists important stuff like the Size of
the heap – so in our case, though we have a 2 GB heap the retained size after
GC is 473.1 MB.
The graph of “Biggest Objects by Retained
Size” will give you immediate clues and this is also reflected in tabular
format in the “Dominator Tree” report.
|
Overview
But the best thing about MAT is the Leak
Suspects report which points out clearly potential memory leaks.
|
So in our hprof, 55% of the heap was being
retained by one instance of weblogic.xml.query.xdbc.Context
i.e. if this object was reclaimed by GC, we could get back 273 MB of memory.
This object was residing at memory location weblogic.xml.query.xdbc.Context @
0xb2477930
The important thing to see in this view is
the Retained Heap
In Memory Analyzer the term shallow size
means the size of an object itself, without counting and accumulating the size
of other objects referenced from it.
The retained heap is the total objects and
memory which this object is holding onto.
We took the
next snapshot after 2 hours of the first heap dump and it showed the same
instance of weblogic.xml.query.Context@0x7c1642f8
had now grown to 301.6 MB with the total heap now at 524.7 MB
Our last
snapshot in confirming our problem was after a further 2 hours, so a total of 4
hours since the first heap dump showed this object.
The heap
had now grown to 616.5 MB with 54% of it occupied by the same object
So next we
now found out from MAT what was causing the leak. We next had to analyze how this class weblogic.xml.query.xdbc.Context was being used in the app and how we could prevent the leak. I will post that in Part 3.