As part of
the regular pre-Production testing it is common to conduct Performance tests
for load under Peak and Soak conditions.
So we had
this Weblogic platform on which all Peak tests were good and so also were the
12 hour Soak tests.
The JVM
memory usage graph plotted showed regular GC clearing up memory and no obvious
leaks. A simple graph plotted in Excel showing JVM Used Heap over time is shown
below. It shows the pattern fairly close to an ideal sawtooth.
An additional
test planned was an Extended Soak – mainly letting the system run as per
expected normal volumes over 7 or 10 days to see if the JVM would throw up any
unusual memory usage patterns or memory leaks.
And we came across some interesting issues !!
We did not
have much of a problem for the first 24 and 48 hours – but after 48 hours (2
whole days) of continuous load, the graph showed that the JVM was unable to clear up any memory even after
GC.
As the
graphs above show the Weblogic server JVM inspite of multiple GCs was not
reducing the utilized memory down to the usual 30 - 40% or so. It was staying
at 80% or so until after 7 days the servers went OutOfMemory and just crashed.
So in order
to analyze this we decided to take JVM Heap Dumps and analyze this using
VisualVM or Eclipse MAT.
In this
instance Eclipse MAT which is an easy Eclipse plugin gave an instant feedback and we
were able to narrow down on Leak Suspects and actually find the root cause of
the problem.
I will
elaborate on that in Part 2 of this series.
