You Never Know

Some info on Weblogic clustering

2010-10-09T00:30:00.001-07:00

Recently this question regarding Weblogic clustering came up on Stackoverflow.

I put some time into researching the answer I provided, and thought of putting this up on the blog as well.

Question from user Russell

I've read http://download.oracle.com/docs/cd/E11035_01/wls100/cluster/overview.html and searched this topic on the internet but still had a hard time understanding some of weblogic's clustering concepts.

Can anybody confirm/correct my understandings below?

•a cluster contains one or more logical servers which can reside on one or many physical servers
•when deploying a j2ee app to a cluster, it is tied to one server in that cluster
•external users of the deployed app aren't aware of clustering
•the log file of that app is located on the server it's deployed
•if the server hosting the app fails, it's okay because the app is in a cluster and another server will pick up the work?
•if the server hosting the app fails, what happens to logging?
Maybe I got the whole concept wrong. Could anybody point me into the correct directions?

Answer

It's good to understand the concept of Domain first.

The Domain is the parent of a Cluster. It contains typically one Admin and one or more Managed servers.
Now the Cluster is a grouping of some or all of these managed servers within the domain.

This diagram here should help understanding the relation between Domains and Clusters

Once you configure a Domain and a Cluster yourself on a development environment, you'll get to know more about it.

Now here are the answers to your specific questions

•a cluster contains one or more
logical servers which can reside on
one or many physical servers

True. But let's clarify what you mean by 'logical' servers.
In the Cluster you typically have two or more Managed servers. These servers run in their own JVMs and can be started independently and serve requests independently. Each server will have a unique IP:port address, and it can be directly accessed from the browser. But these server instances can reside over multiple physical servers.

•when deploying a j2ee app to a
cluster, it is tied to one server in
that cluster

No it is not tied to one server. When you deploy a J2EE app to the Cluster, it will get deployed in turn to each server in that cluster. The JNDI is cluster-wide and each server maintains a local copy of the JNDI.

You can look up the object (say an EJB) via JNDI on the Cluster or on the individual server. Also see what types of Objects can be clustered.

•external users of the deployed app
aren't aware of clustering

True.

But in this case you should have an Apache web server or a load balancer or DNS server which takes the request from the browser, and internally maps it to one of the servers in the cluster. If you dont have any of these, you would have to define the cluster address as a DNS name or IP address for the client. See the section "Avoiding Listen Address Problems " on http://download.oracle.com/docs/cd/E13222_01/wls/docs103/cluster/setup.html#wp682940

•the log file of that app is located
on the server it's deployed

True, one weblogic log per server.

•if the server hosting the app fails,
it's okay because the app is in a
cluster and another server will pick
up the work?

Not by default, you have to configure it for failover and replication. This is a huge topic which needs separate reading

•if the server hosting the app fails,
what happens to logging?

Logging stops. You'll see some shutdown or heartbeat errors in the log, or outofmemory or whatever reason for failure. you'll have to restart the server - and logging continues in a new file (depending on your logging settings)

My original answer is here on stackoverflow for those interested.

Weblogic JDK 1.6 Error Could not reserve enough space for object heap

2010-02-17T05:12:00.000-08:00

Weblogic 10 running on HP-UX throws this error on startup when we try to give 3 GB Xms and Xmx.


-Dweblogic.system.BootIdentityFile=/portalapp/bea/user_projects/domains/MSD4/servers/MS4/data/nodemanager/boot.properties -Dweblogic.nodemanager.ServiceEnabled=true -Dweblogic.security.SSL.ignoreHostnameVerification=false -Dweblogic.ReverseDNSAllowed=false -server -Xms3g -Xmx3g -Xmn1024m -XX:+AggressiveHeap -XX:+UseParallelGC -XX:ParallelGCThreads=20 -XX:PermSize=1024m -XX:MaxPermSize=1024m -XX:MaxTenuringThreshold=3 -XX:LargePageSizeInBytes=4m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:SurvivorRatio=8 -XX:-UseAdaptiveSizePolicy -Xloggc:/portal_logs/gclog/MS4_gclog.log weblogic.Server >

<Feb 17, 2010 10:56:06 AM> <Info> <NodeManager> <Working directory is "/portalapp/bea/user_projects/domains/MSD4">

<Feb 17, 2010 10:56:06 AM> <Info> <NodeManager> <Server output log file is "/portalapp/bea/user_projects/domains/MSD4/servers/MS4/logs/MS4.out">
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.
<Feb 17, 2010 10:56:06 AM> <Debug> <NodeManager> <Waiting for the process to die: 13663>
<Feb 17, 2010 10:56:06 AM> <Info> <NodeManager> <Server failed during startup so will not be restarted>

Additional information:

os.name = HP-UX
os.version = B.11.31

java.version = 1.6.0.04
java.vm.info = mixed mode
java.vm.name = Java HotSpot(TM) Server VM
java.vm.specification.name = Java Virtual Machine Specification
java.vm.specification.vendor = Sun Microsystems Inc.
java.vm.specification.version = 1.0
java.vm.vendor = "Hewlett-Packard Company"
java.vm.version = 11.3-b02-jre1.6.0.04-rc2
java.runtime.version = 1.6.0.04-jinteg_28_apr_2009_04_46-b00

Analysis:

Though we are using a 64-bit server which has 16 GB RAM, we are unable to assign more than 2 Gb to the Weblogic heap.

This value in the startup log shows that the JVM is a 32-bit one.

sun.arch.data.model = 32

In a 32-bit JVM – the process cannot address memory higher than 4 GB in all. But as per Sun site, in reality we cannot get more than 2G.

As per this URL: http://java.sun.com/docs/hotspot/HotSpotFAQ.html#gc_heap_32bit

Why can't I get a larger heap with the 32-bit JVM?

The maximum theoretical heap limit for the 32-bit JVM is 4G. Due to various additional constraints such as available swap, kernel address space usage, memory fragmentation, and VM overhead, in practice the limit can be much lower. On most modern 32-bit Windows systems the maximum heap size will range from 1.4G to 1.6G. On 32-bit Solaris kernels the address space is limited to 2G.

So we tuned the heap to 2 GB for now. i.e. set Xmx and Xms to 2G and that solved it.

More tuning tips on slow Weblogic 10

2010-02-12T22:05:00.000-08:00

In the earlier parts of this article I described some technical tuning tips specific to a slow-performing Weblogic JEE server.

Another earlier post has looked at how to set the JSP check seconds values.

This is a follow-up article specifically for tuning the similar settings for Weblogic 10 - since the syntax has changed.

In weblogic.xml, ensure the following are set as below:

<wls:session-descriptor>
<wls:encode-session-id-in-query-params>true</wls:encode-session-id-in-query-params>
<wls:cookie-comment>mysessioncookie</wls:cookie-comment>
<wls:cookie-max-age-secs>-1</wls:cookie-max-age-secs>
<wls:cookie-name>JSESSIONID</wls:cookie-name>
<wls:cookie-path>/</wls:cookie-path>
<wls:cookies-enabled>true</wls:cookies-enabled>
<wls:timeout-secs>1200</timeout-secs>
<wls:cookie-domain>@MY_COOKIE_DOMAIN@</wls:cookie-domain>
</wls:session-descriptor>

Assuming that no JSPs are being hot deployed or dropped directly into the production server

<wls:jsp-descriptor>
<wls:keepgenerated>true</wls:keepgenerated>
<wls:precompile>true</wls:precompile>
<wls:page-check-seconds>-1</wls:page-check-seconds>
</wls:jsp-descriptor>

and this one

<wls:container-descriptor>
<wls:servlet-reload-check-secs>-1</wls:servlet-reload-check-secs>
</wls:container-descriptor>

Ensure all of these are inserted at the right place within the weblogic.xml to avoid any start up error messages

Check against the reference here http://download.oracle.com/docs/cd/E12840_01/wls/docs103/webapp/weblogic_xml.html

Postcards from New Zealand

2010-01-23T20:27:00.001-08:00

I spent 2 months on a client assignment in New Zealand last year. The summer was a great time to be there and here are some pictures.

Auckland skyline

Looking up at the SkyCity Tower Auckland

Cathedral Cove in the North Island

View of Auckland from Waiheke Island

The Cruise on Milford Sound in the South Island. The view ahead is the Trans Tasman Sea.

The seals at Milford Sound

Birds at Queenstown

Lake Pukaki on the way to Mount Cook

View of Mount Cook from Lake Tekapo

The Church of Good Shephed on the shores of Lake Tekapo

View of Lake Tekapo from Mount John Observatory

Two men playing chess at Christchurch City Centre. Let me know if it's you :)

Weblogic JMS Performance Tuning Tips

2009-08-06T05:04:00.000-07:00

Here are a couple of real-life tips on tuning JMS performance.

Problem:
The creates and sends JMS messages on an outgoing queue for consumption by another application.

It was observed that when the consuming application was offline for a period of time the number of messages that could be retained on the queue before the JVM heap was filled up was quite low. This was tested to be roughly 7000 messages, after which OutOfMemory exceptions begin to occur. Given that the consuming application could realistically be offline for a period, hence the use of an asynchronous queue, we needed to increase the number of messages that could realistically be stored in the queue.

The exception we get is shown below:


Start server side stack trace:
java.lang.OutOfMemoryError:

Start server side stack trace:
java.lang.OutOfMemoryError
        <<no stack trace available>>
End  server side stack trace
        at weblogic.rmi.internal.BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:109)
       at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:127)
       at weblogic.jms.dispatcher.DispatcherImpl_WLStub.dispatchSyncFuture(Unknown Source)
       at weblogic.jms.dispatcher.DispatcherWrapperState.dispatchSync(DispatcherWrapperState.java:286)
       at weblogic.jms.client.JMSSession.createProducer(JMSSession.java:1484)
       at weblogic.jms.client.JMSSession.createSender(JMSSession.java:1335)
...

The GC logs also show frequent Full GCs before the server goes out of memory.

Solution Steps:

1. Enabling JMS Paging

Paging had not been enabled for the queue. Despite this queue being persistent, this meant that every message was stored in the JVM memory heap in its entirety. Enabling message paging for this queue means that only the headers for paged messages are kept in memory, significantly reducing the amount heap utilized.

As the messages were being persisted via a JDBCStore to a database, this functioned as a paging store as well, however a FileStore must still be specified as the paging store for the JMS Server, or the JMS Server will not deploy at WLS server start time. This would be due to the need to cater for any non-persistent destinations when paging is enabled. If non-persistent messages are not paged, the size of this FileStore will be negligible. Paged messages still occupy some space on the memory heap as the message headers are still kept in memory.

On enabling paging for the specific queue, the test could cater for roughly 15000 messages before OutOfMemory exceptions occurred. The point at which paging began was set deliberately low to 100. Recovering from the page store does incur a certain performance cost, so in Production this was set to a more reasonable number based on the peak number of messages expected in the queue under normal conditions.

Despite the gain in number of messages that could be catered for, the heap utilisation graphs were very similar to those before paging was enabled. This showed no minor GCs, only full GCs at fairly frequent intervals.

2. JVM Settings and Garbage Collection Tuning

The untuned JVM heap size was 512Mb. This value could be increased, but test results after tuning the JVM settings indicate that this was probably more than adequate.

Examining the current JVM settings uncovered some settings that needed to be changed.

The most significant issue with the JVM settings was the NewSize value. This was set very high to 384Mb out of the total heap of 512 Mb.
A reasonable New Generation area would normally be 20-25% of the total heap size and setting it larger than the Tenured Generation area is guaranteed to cause unhealthy GC operations. In addition, it is good practice to use NewRatio rather than NewSize to avoid fixing an absolute size. A NewRatio of 3 (1:3, i.e. 25% of heap) or 4 is considered the most appropriate for WebLogic Server applications. NewSize was therefore dropped and a NewRatio was set to 4 (20% of total heap).

The SurvivorRatio value was set to a reasonable value of 3.
TargetSurvivorRatio, however, was unset meaning that the default of 50 applies. 80 would probably be a better setting, meaning that the switch between survivor spaces in the JVM heap would occur at 80% rather than 50%. The performance improvement from this should be noticeable in the frequency of minor GC, though not dramatic.

The PermSize values were high, with PermSize and MaxPermSize both set to 384Mb. These were reduced to 64Mb and 128Mb respectively, which should be more than adequate. Though these changes are unlikely to improve performance, having the PermSize set to high will needlessly consume memory.

The effect of setting a good NewRatio value was dramatic. Many minor GCs were the rule, with infrequent full GCs. 60,000 messages were added to the queue before the heap was approaching full. We ran an overnight, and somewhere between 60,000 and 70,000 messages the OutOfMemory exceptions occurred.

Scaling this up to the Production environment which has 1Gb Heap, shows that the server could easily cater for the expected load.

Dave Matthews Band - Big Whiskey and the Groogrux King

2009-07-20T01:45:00.000-07:00

My favourite group, Dave Matthews Band released their 7th studio album - "Big Whiskey and the Groogrux King". I feel this is their best album since 1998's "Before These Crowded Streets" with them going back to their jamming style of music rather than the manufactured pop sounds of recent albums.
The name is a tribute to founding member and saxophonist LeRoi Moore who died in August 2008 and his nickname was Groogrux. A few of the songs on the album have references to LeRoi and show that the band really miss their mate. The cover portrays Moore's head on a float in the Mardi Gras procession in New Orleans.

The album begins with a short and lovely saxphone solo "Grux" - recorded by LeRoi before his death.

This is the prelude to the first song "Shake Me Like a Monkey" which is one of the outstanding tracks. The lyrics exhort a babe so hot "that God and the devil alone could not have made you up. Two must have worked as one" to "Love Me Baby Shake Me Like a Monkey". and this gem "I like my coffee with toast and jelly. But I'd rather be licking from your back to your belly". Very peppy and a kick-ass blast this song is.

"Funny the Way It Is" is a poem showing the same events has opposite effects on different people. "Hear the laughter while the children play war" and "Funny the way it is, If you think about it, One kid walks ten miles to school, another's dropping out"
The words remind me of Alanis Morissette's "Ironic" and the tune is simple and catchy with subtle violin at the start and a full blooded violin interlude in the middle with strong rock guitar riffs. I think this song also shows the band misses LeRoi with this line "Somebody's heart is broken and it becomes your favorite song". I dont like the video too much though.

The third song "Lying in the Hands of God" is a slow ballad with a simple arrangement. Hummable with reflective lyrics. This song grows on you with it's slow guitar rounding off - and you can picture yourself looking into a fading sunset with a drink in hand.

Then there's another decibel upping blast "Why I Am" - starts off with a full-on drums and guitar rolls and is definitely again a tibute to LeRoi "Why I am still here dancing with the GrooGrux King, We’ll be drinking Big Whiskey while we dance and sing" and you do want to tap along with the dance. This is surely an instant hit.

"Dive In" is my 2nd favourite song on the album - the one I loved as soon I heard it. It starts on a slow note but Dave sounds hypnotical in the lilting tune singing an ode to all things Summer - Summer games, Summer breeze, Summer girls, Summer dresses - "Kick off your shoes, dive in the empty ocean". "Wake up sleepy head, I think the suns a little brighter today". It surely is guys - after listening to this - my day has brightened. Wake up and enjoy it while it lasts cause it wont last forever - is the message here.

"Spaceman" is an okay song compared to the others. If it were on one of the weaker earlier albums, Busted Stuff [2002] or Stand Up [2005], this might have been rated higher but for me, it just makes up the numbers on this album. the lyrics are about a guy's love for his girl, even when things dont work out.

"Squirm" is quite unusual and surprising. It starts off slow but punches you when Dave gets a bit menacing and the crescendo builds as he sings "Drumbeats louder (x 4)". He gets ferocious on the "Open up your head" chorus. Parts of it almost feels like a song by U2, I can imagine Bono cooing in falsetto between the climactic violin and drum rolls, and it blends in what sounds quite Middle Eastern wailing for the finale. There's much more meaning to these lyrics with it's referring to (lack of) Faith and the differences in Religions and the inevitable End. But I love the song for it's music rather than it's philosophy.

"Alligator Pie (Cockadile)" is my favourite song on the album. It's arranged chaos is sublime - listen closely for the dog barking just before the violin in the beginning and Dave's own howling bringing in the vocals. This song will get your feet tapping with the banjo driving the entire melody. On the face of it, this is a song for his daughter Stella asking "Daddy when you going to put me in a song" since many earlier songs have referenced her twin sister Grace. (Yup, even I dont get why didnt her put her in a song earlier) But Dave's also pushes in a critical reference to the Bush administration's response to the Hurrican Katrina devestation of New Orleans. The song proceeds into a lovely cacophony with Dave almost yelping through the words. My only complaint is it ended too soon at 3:57 mins. The title word "cockadile" is the child's pronunciation of "crocodile" but I did not hear the word used in the song. Must be an in-family joke.

"Seven" begins with a heavy guitar sound almost like Pearl Jam and then moves into another jamming event. Most of the song is in falsetto with plenty of naughty lyrics. I wonder if the title itself alludes to The Seven Minutes- the novel by Irving Wallace. But it's fun to hear this all the same - and you'll start humming it in no time.

"Time Bomb" begins so similar to "The Dreaming Tree" from "Before These Crowded Streets". Soft alto saxophone and ballady tune "I'm a ticking time bomb Waiting to blow my top" and - Woah it explodes into a powerful metal guitar as if Metallica guested on this after 2:40 mins. Dave's shouting out angrily and covered by a strong trumpet "I want to pick up the pieces, Hammer in the final nail, I wanna believe in Jesus". Yes this is a dark song about death and killing and the purpose of it all. This song echoes very strongly in your heart if you have experienced loss - either by death or a failed relationship.
I feel the subtext here is on the existence of God and whether we are controlled by a higher power "If Martians fell from the sky what would that do to God? ".

When you're recovering from that rant, the soft guitar chimes in like a refreshing glass of chilled water and thats the opening of "My Baby Blue". This is the best ballad on the album - a love song dedicated to the one true love who he seems to have lost "Holding tight to the legs of all your angels, Goodbye my love ". Talking about the blue eyes, I feel though this is about loss, it's not dedicated to LeRoi, and there are some forums suggesting this is about his sister's death. In a way this seems like he is at peace with the loss that was experienced in "Time Bomb". This is an 'easy listening' track for Pop fans.

"You and Me" is another simple and easy ballad. This is probably a dedication to his wife - "When the kids are old enough we're gonna teach them how to fly. You and me together, We can do anything baby". This is mostly in acoustic guitar and rounds off the album on a soft note.

Yes - This is a lovely collection and took some time coming.

Cheers to Dave, LeRoi, Stefan Lessard, Carter Beauford, Boyd Tinsley, Jeff Coffin and Tim Reynolds with a Big Whiskey in hand.

JVM Tuning from the Trenches

2009-07-13T22:53:00.000-07:00

This article is a follow-up to http://jojovedder.blogspot.com/2009/06/slow-weblogic-part-6-jvm-heap-analysis.html. Please read that one first for the basics on JVM heap, parameters and flags.

Problem:
Platform running Weblogic 8.1 on Sun V880 servers. Total RAM of 32 Gb on the machine.
2 Gb assigned to the managed server JVM heap. JDK 1.4

Initial settings:


-XX:+AggressiveHeap -Xms2048m -Xmx2048m  -XX:SurvivorRatio=32 -XX:MaxPermSize=128m

But still there are 20 Full GCs per hour in peak times, before the server crashes.

Analysis

1. It was decided to reduce the SurvivorRatio to 4 and restart with some more flags.

The size of ONE Survivor Space is calculated as

SurvivorSpace = NewSize / (SurvivorRatio + 2)

Keeping SurvivorRatio as 32 means the Survivor spaces are too small for promoting stuff from Eden. Hence we reduce this to 4 which allows for larger Survivor spaces.

2. As per Sun Bug ID: 6218833, setting AggressiveHeap set before Heapsize (Xmx and Xms) can confuse the JVM. Revert the order to have -Xms and -Xmx to come before -XX:+AggressiveHeap or not use it

3. The application has 180+ EJBs with pools of beans. Hence set the -Dsun.rmi.dgc.client.gcInterval=3600000 (1 hour) instead of the default 60000 (1 min). More on this here: http://docs.sun.com/source/817-2180-10/pt_chap5.html

4. The site is restarted once a week at 4:30AM. The patterns stays normal for 2 days – and then degrades into full GC.

5. The Old space is pretty much full – at every minor collection – the Old space must be cleared up for promotion from Young to Old to take place.

6. Permanent space is pretty much full – keeps loading classes and classes ( could that be a problem – the difference between the number of JSP’s per Release?)
Hence we increased the PermSpace from 128M to 256M

7. Ensure we are running the server JVM by using the -server flag

8. Use OptimizeIt or similar profiling tool to see the memory usage and find code bottlenecks.

The settings now were


-server -Xms2048m -Xmx2048m  -XX:MaxNewSize=512m -XX:NewSize=512m -XX:SurvivorRatio=4 -XX:MaxPermSize=256m -Xincgc -XX:+DisableExplicitGC -XX:+AggressiveHeap -XX:-OmitStackTraceInFastThrow

This reduced the Full GCs to one a day.

Error Logs

At the time of the server going out of memory prior to a crash, the logs are filled with repeated errors (up to 100 repetitions) of this sort


java.lang.NullPointerException
 <<no stack trace available>>

Adding the -XX:-OmitStackTraceInFastThrow flag resolves this problem, the root cause of the NPE it self has to be tracked down but we do not have any longer the issue of huge recursive exception strings.

We could now see the stack trace as


java.lang.NullPointerException
 at java.util.StringTokenizer.(StringTokenizer.java:117)
 at java.util.StringTokenizer.(StringTokenizer.java:133)
 at jsp_servlet._framework._security.__login._jspService(login.jsp:294)
 at weblogic.servlet.jsp.JspBase.service(JspBase.java:27)
 at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1075)

This seems to be a Sun bug described here.

More on Weblogic Muxers

2009-07-07T02:12:00.000-07:00

This is a follow-up to an earlier article which detailed the Weblogic Socket Muxer pattern in thread dumps.

Another ERROR pattern associated with Muxers is shown below


####<Jun 23, 2009 9:25:24 AM BST> <Error> <Socket> <d001> <managed1> <ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> 
 <BEA-000403> <IOException occurred on socket: Socket[addr=/10.23.22.1,port=1950,localport=61002]
 java.net.SocketException: Connection refused.
java.net.SocketException: Connection refused
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:672)
        at weblogic.socket.SocketMuxer.readReadySocket(SocketMuxer.java:648)
        at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:123)
        at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
        at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
        at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)

As written in an earlier post, each incoming request on a TCP socket uses one File Descriptor.

We monitored the File Descriptors available to this Weblogic server using



Here the Weblogic pid = 8359

/usr/proc/bin/pfiles 8359 | grep rlimit


1024

ls /proc/8359/fd | wc –l


140

The first command shows that 1024 FDs are allocated to this process, and the second shows how many are being used in real-time.

A plot of the second value during high load times shows the FDs go up to 900+ in use and hence a resolution used in this case is to increase the number of FDs.

A recommended value is to increase it to 8192.

Slow Weblogic Part 6 - JVM Heap Analysis using GCViewer

2009-06-15T01:04:00.000-07:00

Basics

In an earlier article, I had listed the review of JVM memory parameters as one of the important checks for tuning the JEE server platform.

The basic primer for JDK 1.4 is at http://java.sun.com/docs/hotspot/gc1.4.2/

The key points you need to know are:


Total JVM Heap = Young + Tenured(also called Old)
Young = Eden + From (SS1) + To (SS2)

In the diagram below [taken from the Sun website], "From" and "To" are the names of the two Survivor Spaces (SS) within the "Young".

Perm Space (and code cache): stores JVM’s own stuff. This is outside the Heap you assign using Xms and Xmx. A good explanation of this is available here

The JVM Heap is at default initial 2Mb and max 64Mb (for JDK 1.4 on Solaris).
Default Perm Size is 16MB (for JDK 1.4 on Solaris)
The defaults change for each JDK and are different on each OS - so look up the values on the respective websites.

The ratios are as shown below

Now the object life cycle and garbage collection occurs like this:

1. Objects when created are always first allocated to Eden.
2. When Eden fills up, a fast but not comprehensive GC (minor collection) is run over the young generation only.
3. All surviving objects are moved from Eden into one Survivor Space.
4. In consequent minor collections, new objects move from Eden into the other Survivor Space, plus everything from the first Survivor Space (survivors from the previous minor collection) is also moved into the second Survivor Space. Thus one survivor should be empty at that time.
5. When objects in Survivor Space are old enough (or survivor fills up), they are moved to Tenured. By default the long-lived objects may be copied up to 31 times between the Survivor Spaces before they are finally promoted to the Old generation.
6. When tenured fills up, a Full GC collection is run that is comprehensive: the entire heap is analyzed, all objects that can be destroyed are killed and memory is reclaimed.

Note: the above lifecycle changes slightly when advanced options such as ConcurrentMarkSweep etc are enabled.

Look Closer

What do these values mean ?

A full list of options available at http://java.sun.com/docs/hotspot/VMOptions.html and http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp

The absolute basic ones are listed in the table below. Note: This is for JDK 1.4
Some of these have changed in JDK 1.6

-Xms1536m -Xmx1536m	These represent the total heap (minus Perm space). Xms is the Initial Heap, set to 1.5Gb in this case. Xmx is Max Heap. It is good practice to set Xms = Xmx The max heap is limited by the RAM available on the server
-XX:NewSize=512m	This specifies the initial size of the Young generation,set to 512Mbin this example. It is better to set this as a percentage of the Heap using -XX:NewRatio
-XX:MaxNewSize=512m	This specifies the maximum size of the Young generation,set to 512Mbin this example. It is better to set this as a percentage of the Heap using -XX:MaxNewRatio
-XX:PermSize=64m -XX:MaxPermSize=128m	These values are the Minimum and Maximum sizes of the permanent generation heap space. Optimally, set PermSize equal to MaxPermSize to avoid heap having to be adjusted when permanent area grows. As specified earlier, this area of memory is over and above the Total Heap set using Xms
-XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90	The New generation area is divided into three sub-areas: Eden, and two survivor spaces that are equal in size. Use the -XX:SurvivorRatio=X option to configure the ratio of the Eden/survivor space size. In the above example, setting it to 8 means the ratio of Eden:SS1:SS2 is 8:1:1. So for a NewSize of 512 Mb, the two SS will be 51 Mb each, and Eden will be 512 MINUS (51 + 51) = 410 Mb. TargetSurvivorRatio of 90 allows 90% of the survivor spaces to be occupied instead of the default 50%, allowing better utilization of the survivor space memory.
-XX:MaxTenuringThreshold=10	This switch determines how many times the objects are hopped between the Survivor spaces before getting promoted to the older generation. The default value is 31.
-XX:+DisableExplicitGC
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -Xloggc:/log/gc.log	These are GC specific settings asking for GC details and the log file name in which these details should be captured

If you have an appetite for more, read this http://java.sun.com/performance/reference/whitepapers/tuning.html

GCViewer

This link below explains how to download and use the GCViewer tool. This is quite a useful tool for viewing the number of GCs and Full GCs and how the JVMis behaving over time.

http://www.javaperformancetuning.com/tools/gcviewer/index.shtml

The most important things to look at in the GCViewer analysis are the

* Acc Pauses - Accumulated Pause Time (total time app was stopped for GC).Pauses are the times when an application appears unresponsive because garbage collection is occurring
* Total Time - Total Time the application runs.
* Throughput - Time the application runs and is not busy with GC. Greater than 99% is fantastic.Throughput is the percentage of total time not spent in garbage collection, considered over long periods of time.
* Footprint - Overall Memory Consumption - Ideally as low as possible. This is the working set of a process, measured in pages and cache lines. On systems with limited physical memory or many processes, footprint may dictate scalability. Thus this usually reflects the size of total Heap allocated via Xms and Xmx

This diagram is taken from the above site:

Tuning Example From the Trenches - Frequent GC due to Perm Space getting Full

JVM Parameters already set

java -server -Xms1024m -Xmx1024m -XX:MaxPermSize=340m -XX:NewSize=340m -XX:MaxNewSize=340m
-XX:SurvivorRatio=9 -XX:TargetSurvivorRatio=90 -XX:+UseParNewGC
-Xloggc:/wls_domains/gclog/jms.gc -XX:+PrintGCDetails -XX:+UseParNewGC
-XX:+PrintGCTimeStamps -XX:+DisableExplicitGC -XX:+PrintTenuringDistribution
-XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime
-XX:+JavaMonitorsInStackTrace -Dweblogic.system.BootIdentityFile=/wls_domains/xx/boot.properties
-Dweblogic.management.server=http://xx.xx.xx.xx:6000 -Dweblogic.Name=xxxxxx
-Dweblogic.ProductionModeEnabled=true
-Djava.security.policy=/opt/bea/wls/8.1sp4/weblogic81/server/lib/weblogic.policy
weblogic.Server

Full GC Pattern

The GC log shows


0.000: [Full GC 0.000: [Tenured: 0K->9794K(700416K), 0.8050952 secs] 134607K->9794K(1016960K), [Perm
: 20479K->20479K(20480K)], 0.8053527 secs]

First Full GC happens at 0:00 sec from start of server. Before and after '->' figures represent size of live objects before and after GC. Number in parenthesis indicates total available space. So from above numbers - Tenured was not full but Perm was almost full.


6579.013: [Full GC 6579.013: [Tenured: 9794K->18941K(700416K), 0.9677233 secs]
155600K->18941K(1016960K), [Perm : 24575K->24575K(24576K)], 0.9679896 secs]

Same thing happens here. The second Full GC took place at 6579.013 sec (1hr 49mins) from startup of server.
Again a Full GC is triggered, but the Tenured was not full. The Tenured is now 18.9 Mb out of 700 Mb - but it is seen the Perm Space has grown to 24.5 Mb and is not getting cleared.


9363.515: [Full GC 9363.516: [Tenured: 18941K->19463K(700416K), 0.6532332 secs] 36950K->19463K(1016960K), [Perm : 28672K->26462K(28672K)], 0.6536095 secs]

At the 3rd Full GC at 9363 seconds after server startup, the Perm space grew to 28.6 Mb and recovered marginally to 26.4 Mb.

Observing this over a long period of time, we concluded that at startup around 20MB is allocated to perm which with each Full GC keeps growing till 30MB and later shrinks back to 25MB and the cycle continues.

The pattern is highlighted below:


0.000: [Full GC 0.000: [Tenured: 0K->9794K(700416K), 0.8050952 secs] 134607K->9794K(1016960K), [Perm : 20479K->20479K(20480K)], 0.8053527 secs]
6579.013: [Full GC 6579.013: [Tenured: 9794K->18941K(700416K), 0.9677233 secs] 155600K->18941K(1016960K), [Perm : 24575K->24575K(24576K)], 0.9679896 secs]
9363.515: [Full GC 9363.516: [Tenured: 18941K->19463K(700416K), 0.6532332 secs] 36950K->19463K(1016960K), [Perm : 28672K->26462K(28672K)], 0.6536095 secs]
13483.233: [Full GC 13483.233: [Tenured: 19463K->16962K(700416K), 0.9783693 secs] 26678K->16962K(1016960K), [Perm : 30719K->21330K(30720K)], 0.9857390 secs]
17308.829: [Full GC 17308.830: [Tenured: 16962K->17312K(700416K), 1.0578872 secs] 88025K->17312K(1016960K), [Perm : 25600K->25600K(25600K)], 1.0581738 secs]
21237.810: [Full GC 21237.810: [Tenured: 17312K->17814K(700416K), 1.4728764 secs] 302290K->17814K(1016960K), [Perm : 29695K->26719K(29696K)], 1.4801234 secs]
30079.672: [Full GC 30079.672: [Tenured: 17814K->18676K(700416K), 1.0282446 secs] 83159K->18676K(1016960K), [Perm : 30975K->27564K(30976K)], 1.0349869 secs]

Solution:
Though the MaxPermSize=340m is provided, initial available Perm Size is getting full and JVM is invoking a Full GC to free up memory. The default initial PermSize is 16Mb and hence the Perm Space is resizing itself as the JVM grows.

Not sure if this behaviour is a bug but by adding an initial PermSize of 64K using this flag -XX:PermSize=64m resolved this issue of frequent Full GC.

UPDATE: Another examples published on http://jojovedder.blogspot.com/2009/07/jvm-tuning-from-trenches.html

Weblogic - Socket Muxers in Thread Dumps

2009-05-17T22:47:00.000-07:00

What are these weblogic.socket.Muxer threads seen in thread dumps ?

Note: for a basic primer on taking thread dumps and analyzing them, see this earlier article

Socket Reader Threads accept the incoming request from the Listen Thread Queue and put it on the Execute Thread Queue.

In WL 8.1, there are 3 socket reader threads by default.
In WL 9 and 10, WebLogic allocates 33% of server threads to act as socket readers by default. This need not be changed usually.

One socket reader thread is usually in the poll function, while the others are available to process requests.
The polling thread is highlighted in the thread dump below.


"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x016b2148 nid=0x42 waiting for monitor entry [5997f000..5997fc28]
at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:91)
- waiting to lock <0x94846b40> (a java.lang.String)
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)


"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x00683c28 nid=0x41 waiting for monitor entry [59a7f000..59a7fc28]
at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:91)
- waiting to lock <0x94846b40> (a java.lang.String)
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)


"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x0079e5b0 nid=0x40 runnable [59b7f000..59b7fc28]
at weblogic.socket.PosixSocketMuxer.poll(Native Method)
at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:100)
- locked <0x94846b40> (a java.lang.String)
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)

In an earlier support case on Stuck Threads, we asked BEA:

Should we worry about the Weblogic.socket.Muxer threads which always show 2 threads waiting for lock and 3rd thread locking the same object?

The Muxer TD is attached. This shows same behaviour on all our Weblogic servers.


Full thread dump Java HotSpot(TM) Server VM (1.4.2_05-b04 mixed mode):

"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x0151
c588 nid=0x1b4 waiting for monitor entry [ad57f000..ad57fc28]
        at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:91)
        - waiting to lock <0xd9331760> (a java.lang.String)
        at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
        at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
        at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)



"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x0161
d608 nid=0x1b3 runnable [ad67f000..ad67fc28]
        at weblogic.socket.PosixSocketMuxer.poll(Native Method)
        at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:100)
        - locked <0xd9331760> (a java.lang.String)
        at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
        at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
        at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)



"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x01bb
6730 nid=0x1b2 waiting for monitor entry [ad77f000..ad77fc28]
        at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:91)
        - waiting to lock <0xd9331760> (a java.lang.String)
        at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
        at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
        at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)

The reply from BEA Support was that the above pattern of weblogic.socket.Muxer threads are not a cause of stuck threads.

Why do they mostly show as being Stuck in Samurai TD analyzer ?

As the image shows, when you analyze thread dumps using Samurai, the muxer threads are shown as being Stuck since they're all locked on the same object. This is probably treated as a deadlock condition.


"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'"
- waiting to lock <0xd9b61098> (a java.lang.String)

"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'"
- waiting to lock <0xd9b61098> (a java.lang.String)

"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'"
- locked <0xd9b61098> (a java.lang.String)

But you will see the same in any Thread dump even on a development instance with no requests.
The locks mentioned do show up as red in Samurai - but they aren't deadlocks just regular locks.

A thread gains an exclusive lock on an object to perform some action, then frees it allowing the next thread to gain access.

Additionally, if you look at the thread dumps over time, you'll see that these specific locks are not always present - they are moving between the threads which is indicative of their transitory nature.

I want to know more details on Muxers

The socket Muxer manages the server’s existing socket connections.
It first determines which sockets have incoming requests waiting to be processed. It then reads enough data to determine the protocol and dispatches the socket to an appropriate runtime layer based on the protocol.
In the runtime layer, the socket muxer threads determine which execute thread queue to be used and delegates the request accordingly.

From the documentation on http://edocs.bea.com/wls/docs100/perform/WLSTuning.html#wp1152246 ,
Weblogic has two versions of the socket muxer, one is the Java version and the other uses a native library which makes better use of operating system calls. The Enable Native IO checkbox on the server’s configuration settings tells the server which version to use. This is ON by default for most platforms.

Native muxers provide superior scalability because they implement a non-blocking thread model. When a native muxer is used, the server creates a fixed number of threads dedicated to reading incoming requests. Oracle recommends using the default setting of true for the Enable Native IO parameter which allows the server to automatically select the appropriate muxer to use.

You must ensure that to use Native I/O, the native library must be present in the server’s shared library path . This is set up with the default scripts.
When the server does not find the native library, it throws an error
java.lang.UnsatisfiedLinkError: no muxer in java.library.path
and then loads the Java version of the muxer.

Confirm the LD library path is okay and pointing to the Solaris LD path. Check the startup log when starting a managed server. What is the value of java.library.path?
This is where the JVM actually get's the library from.

http://m-button.blogspot.com/2008/08/how-does-weblogic-handle-socket-muxers.html has a good example of how to identify Native vs Java muxer in a thread dump.

The Thread Dump I’ve used in my examples above uses the Native muxer (weblogic.socket.PosixSocketMuxer) on Solaris.

Solaris has another Native muxer called the weblogic.socket.DevPollSocketMuxer
An example TD using this muxer is shown below.


"ExecuteThread: '4' for queue: 'weblogic.socket.Muxer'" waiting for lock java.lang.String@4edf4f BLOCKED
weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:95)
weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)



"ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" RUNNABLE native
weblogic.socket.DevPollSocketMuxer.doPoll(Native Method)
weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:96)
weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)



weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)
"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" waiting for lock java.lang.String@4edf4f BLOCKED
weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:95)
weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)



"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" waiting for lock java.lang.String@4edf4f BLOCKED
weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:95)
weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)



"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" waiting for lock java.lang.String@4edf4f BLOCKED
weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:95)
weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

To change the number of Muxers from the default, follow the instructions given at http://e-docs.bea.com/wls/docs92/ConsoleHelp/taskhelp/tuning/TuningSocketReaders.html

See http://jojovedder.blogspot.com/2009/07/more-on-weblogic-muxers.htmlfor an update on Muxers

Additionally on Oracle JRockit JVMs - there are some information in the thread dumps which point out the same problem in a different manner.

After the normal stack dumps, BEA JRockit performs a deadlock detection. This is done by finding "lock chains" in the Java application. If a lock chain is found to be circular, the application is considered caught in a deadlock.

A detailed explanation of the 3 types of lock chains in JRockit is given here

What is relevant for us is the example of Muxers which are shown as:

Blocked lock chains
===================
Chain 2:
"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=129 idx=0x218 tid=4079 waiting for java/lang/String@0x37804000 held by:
"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=127 idx=0x210 tid=4077 in chain 1

Open lock chains
================
Chain 1:
"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" id=128 idx=0x214 tid=4078 waiting for java/lang/String@0x37804000 held by:
"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=127 idx=0x210 tid=4077 (active)

As per the explanation, the Open lock chain depicts Thread 1 waiting for Thread 0. This is not a deadlock, only a straight dependency.

Since Thread 0 is already part of the Open lock chain, the fact that Thread 2 is also waiting on the same Thread 0 is treated as a "Blocked lock chain".
In this case this is not a problem.

Update 15th Feb 2011

I'm glad this blog entry on weblogic muxers has made it onto the Oracle forums with a mention from James Bayer.

Slow Server Response Part 4 - Platform Checks and Action Plan

2009-05-12T00:38:00.000-07:00

In the earlier parts of this article I described some technical tuning tips specific to a slow-performing Weblogic JEE server.

Another earlier post has looked at analyzing thread dumps and prstats when the Weblogic/JEE server consumes high CPU.

This article provides a sequence of actions a Support team should carry out when faced with a critical situation - the site is down / servers running high CPU - and management teams want quick updates - and an action plan.

So here is (from experience) the Rapid Action Plan:

Technical Checklist for the Platform

1. Start a diary and mark out each of the steps given below as well as any following action with the timestamp at which it was carried out and the result/status.

2. Get the relevant technical experts on a conference call + Netmeeting/LiveMeeting/Desktop Sharing/Remote Admin.

3. Does the application have any traps or thresholds set which are configured to automatically raise alarms to the Support teams? Have any of the traps set been exceeded ? eg: Server CPU, Memory utilization, No of Threads

4. Can we narrow down to a problem area - Web server, Application server, Database, OS - based on log files, error messages and Support team or User input.

If a particular Weblogic Managed Server is identified as a point of failure, does the Configuration allow that server process to be shutdown for a while - thereby reducing Cluster capability but still providing acceptable Quality of service.

Are all the Managed Servers running with equal threads? If not, this can cause a load balancing issue.

Sometimes the bottleneck can be the Web server plugin which is not able to properly load balance the requests across the Weblogic cluster. This is usually the case when users complain of loss of session, spontaneous logout etc. The problem can be the user has been bounced from one Weblogic server to another in the cluster - and the session might not be replicated across the servers.

Any redundant JDBC connection pools - i.e. those configured with a high capacity but monitoring shows they dont need that many.
Then reduce the Capacity of that pool so that it does not hold on to unnecessary connections on the database.

5. From the log files, identify whether a particular application or code area is causing an issue. eg: EJB throwing errors, Spring bean configuration missing.

6. Are the log files too large (> 500 Mb) or not getting rotated via Weblogic rotation policy or Unix archiving ?

7. Check the Downstream Back-end systems which the server connects to - via DBLink, Web service, XML/Http, JMS, HTTP screen scraping etc.? Any known issues or planned outages? There should be error logs pointing in that direction. Contact their support teams to know if their system is available and returning responses as per the SLA.

8. Can the problem be replicated on Reference/Test instances?
A Dev or Test team can in parallel try out to see whether the issue is replicable.

If Yes, is it code related or configuration related?

If the issue is not replicable, then can it be data related ? Perhaps a particular set of data exists on Production which is not on the Test instance - and that could be the problem. Can the data be brought into Test to try and replicate the problem ?

9. Can it be content related? Does the platform have a Content Management System? Is the link from the CMS to the server working or broken? Is the Content correctly getting deployed into the database + file system as per daily process?

Check if there was a content deployment carried out and any records whether they failed or passed. Is content deployment happening during the business timings, and utilizing system and CPU resources - which chokes the JEE server?

Can a resource-hungry content deployment process be moved to out-of-business hours.

10. Test broken user journeys on the site.

Can the problem be seen while running HttpHeaders, HttpAnalyzer, Fiddler etc ? Does it show any change in HTTP parameters such as Cookies, Session timeouts?
Compare these against the Test environment and see whether any mismatches which could cause the problem.
If there is bouncing of user sessions between managed servers, this will be visible in the weblogic JSessionID which will be different on the client browser.

11. What were the last few changes to the platform ?

Check latest release or configuration change as per Support Team Diary of Events. Could these have caused an issue and should these be rolled back?

Were these properly tested and signed off before going into Production.

eg: any new Database driver, changes to TCP parameters, JTA timeouts increased?

12. Check the last few support cases raised? See if there were any problems reported by business or end customers.

13. Solaris/OS checks

Is the platform running on the latest OS patch levels and JDK settings as recommended by Sun.

a. No of processes running. Use
ps -ef | wc -l

b. Ping the boxes, to check if they are alive

c. CPU utilization

prstat

d. Memory utilization

vmstat 3

Swap space utilization, amount of space in /tmp - is there any old file or core dump occupying the directory used as swap space. We once moved old EAR files from /tmp on the server; memory utilisation went from approx 90% down to 65%.

e. Disk space

df -ek

f. No of File descriptors

14. Weblogic/Web server checks

a. Thread utilization - any Stuck Threads

Analyze Thread dumps , at least 4 sets of Thread dumps taken 5 seconds apart when stuck thread is observed. See here for more details on what to look for in the thread dumps. Use Samurai or TDA

b. CPU %

c. Access and error logs - Any CRITICAL messages in the logs. Any Connection_Refused errors indicating the threads were not able to accept new requests.

d. No of open sockets to weblogic

netstat -a | grep

e. Memory utilization via Weblogic console

f. Check via console if all the managed servers are up and running

g. Connection pool utilization, Are they hitting the peak values.

h. Frequent Garbage collection shown in the console?
Frequency of GC, GC pattern. Has the JVM been tuned to allow optimum garbage collection. See this URL for more.

i. Check for the values in weblogic.xml for jsp-pageCheckSeconds and servlet-reload-check-secs - if these are at the default of 1, the server will check each second to see whether the JSP should be recompiled - this is horribly slow

j. Cron job logs - any failures.

k. No of weblogic sessions per server - the more the number of HttpSessions, the higher the memory (RAM) that gets used.

l. Is a large part of the application journey over SSL. When supporting the cryptography operations in the SSL protocol, WebLogic Server cannot handle as many simultaneous connections.
Typically, for every SSL connection that the server can handle, it can handle three non-SSL connections. SSL reduces the capacity of the server by about 33-50% depending upon the strength of encryption used in the SSL connections.
(Source: http://edocs.bea.com/wlp/docs92/capacityplanning/capacityplanning.html#wp1080286)
Consider reducing the SSL journeys on the site.

m. Disk space taken by Weblogic and other logs such as Log4J.
Is log4j running in DEBUG and writing out loads of logs ? This will also slow down the server horribly.

15. Database checks

a. SQL Server locks (Call out DBA)

b. Database stuck/locked processes

c. Any DB link down

d. Any issues with open cursors, cached cursors ?

e. Is the database running at very high Memory Utilization?

16. Search Engine processing - check the log for the day.

17. Any MIS such as Webtrends / Omniture Analysis - for application usage. Has there been a sudden rise in users on the site - eg a marketing campaign or a new feature gone live - causing a rise in usage which the infrastructure cannot cope with.

18. Any application cached data which was wiped out and took time to rebuild - causing slow service in the interim period. eg: is any database table with a lot of rows being cached.
Or conversely, is there incorrect data in a certain cache and will clearing the cache help ?

19. SMTP email delivery failures due to any problems on the OS ?

20. Any planned backup processes running on the OS which takes up a lot of CPU.

Remedial actions

1. Make a list of possible changes based on the above checks to address these problems.

2. Only change one setting on any system at a time. Test and record the desired effects and observed effects. Be clear on why a particular change is being made.

3. If it doesn't work rework the plan to get to root cause of failure.

4. Be aware that reactive changes will be made directly to the production
environment by various parties. Significant changes will be made purely to enable investigation and diagnosis of issues.
The lack of up-to-date documentation creates risk. Maintain a documented rationale for a particular design decision, configuration choice, or system parameter; this reduces the likelihood that mistakes will be repeated. Documentation is a key communication tool, without it intent may be miscommunicated within the team. If key staff members leave, knowledge will be lost to the extent that the platform may become unmanageable.

5. Add additional tests to the regression test
suite. Increase the coverage of the regression test suite, focussing on
simulating live system interaction.

6. Over the long term, identify and re-architect towards removing Single Points of Failure - such that loss of a single machine or process would not lead to a loss of service.

Examples:
· Single web server machine, hosting the Apache/SunOne instances.
· Single application server machine, hosting the Weblogic application server.
· Single database server instance.

The system runs at risk of lengthy service outage if any one of these components fails. If a hardware failure occurred and one of the servers was lost, alternative hardware would need be installed and initialised from back-up tapes.
This needs to be fixed and stabilized over the long term.

6. While the analysis is going on, a member of Support team should circulate the KEY Metrics on an hourly basis to the TECHNICAL community. Ensure this is the important dataset and not too much info which just becomes noise.

Example in the table below:

Servers	Server 1	Server 2	Server 3
Idle Threads	11	14	17
JMSErrors	1	0	17
IOExceptions	3	0	1
Stuck Threads	0	0	7
JMSErrors	1	0	17
netstat -a \| grep TIME_WAIT \| wc -l	186	289	69
CPU Utilization (%)	12	1.8	3.6
Memory Utilization (%)	3.9	2.9	2.1
500 Internal Server Error	3	0	0

No of logged in Users: 260

JMS Pending Messages: 0

Any queries or clarifications, leave me a comment and I'll try to get back.

Slow Weblogic Response Part 3 - Tuning File Descriptors

2009-05-07T22:02:00.000-07:00

This article is a follow-up to the Overall Tuning considerations discussed earlier.
A very important Operating System value to be tuned are the File Descriptors

File Descriptors and relation to Sockets

A File Descriptor (FD) is a handle created by a process when a file is opened. Each process can use a set limit of FDs and this is usually an OS level setting.
In the Solaris 8 version, the default is 1024. In later Solaris releases, the default is 65,536 but this needs to be set in the /etc/system as given below.
So, the default available for a process on an untuned OS is 1024.


# ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 8192
coredump(blocks) unlimited
nofiles(descriptors) 2048 
memory(kbytes) unlimited

In a JEE server, each incoming request uses a TCP socket and this socket consumes a file descriptor from the total available for the process.

As the number of requests coming into the server increase, you can face a situation where there are many sockets open and thus you run out of FDs
This could happen if you have a large number of clients (~1000 or more). This could also happen if you have HTTP connections with keep-alive turned off and so a lot of sockets are in TIME_WAIT.

If you have Stuck Threads, then those keep the FDs open and those wont be closed until the thread is released.

File Descriptors limit exceeded

A Weblogic server throws the below error when the FD limit has been exhausted.



BEA-000204 java.net.SocketException: Too many open files

OR

java.io.IOException: Too many open files 
    at java.lang.UNIXProcess.forkAndExec(Native Method)

When a Weblogic server starts up, there is an INFO message in the logs stating how many FDs have been allocated to the process.


<<WLS Kernel>> <> <BEA-000415> <System has file descriptor limits of - soft: 2,048, hard: 2,048>

<main> <<WLS Kernel>> <> <BEA-000416> <Using effective file descriptor limit of:

 2,048 open sockets/files.>

To display a process' current file descriptor limit on Solaris, the command is:



/usr/proc/bin/pfiles <pid>  grep rlimit

Example:

$/usr/proc/bin/pfiles 9052  grep rlimit

Current rlimit: 8192 file descriptors

You can check the actual number of FDs being used by a running server at any time using this command:



Note: This is not the max limit but the actual number used by the process at that point of time

ls /proc/<pid>/fd  wc –l

If you monitor this on a regular basis, you can see how often you are nearing the max limit set, and whether the values need to be tuned to support the peak loads and peak traffic timings for your server.

As an example, we plotted FDs used every 5 minutes as below

Time	FD Used
11:00	389
11:05	429
11:10	748
11:15	488
11:20	337
11:25	595

Increasing File Descriptors

The Hard limit is the max value set on the OS.

The Soft limit shows the value set for a particular child process on the OS. This cannot be higher than the hard limit.
You can change the soft limit on-the-fly by using


ulimit -Sn 8192

This will only for that that telnet session and will not be a permanent change. Add this command to the user's .profile file to avoid repeating each time.

If hard limits need to be set, root user needs to update the /etc/system file and the machine to be rebooted. Even though the rlim_fd_max default on Solaris 9 is 65536, it must be in the /etc/system file

The values to be changed are

rlim_fd_max (default hard limit)
rlim_fd_cur (default soft limit)

On Linux:

To increase the hard limit it to (say) 65535, use the following command (as root):

echo "65535" > /proc/sys/fs/file-max

To make this value to survive a system reboot, add it to /etc/sysctl.conf and specify the maximum number of open files permitted:

fs.file-max = 65535

Any queries or clarifications, leave me a comment and I'll try to get back.

Slow Weblogic Response Part 2 - Overall Tuning Considerations

2009-05-06T23:11:00.000-07:00

Part 1 of this article was related to using the JSP precompilation and tuning the recompilation settings provided in Weblogic.

Now, lets take a look at the other parameters that require tuning and setting correctly, in order for the Weblogic server to be generating acceptable performance of your site.

Usually the problem statement is similar to this:

1. The site regularly stops responding and all available execute threads are consumed. All future requests fail to be handled and a server restart is required.
2. The page response times of the entire site are too high and need to be brought down to a more useable level.

Note: An additional post has been published which provides a skeleton Action Plan for analyzing the entire slow site/high CPU issue including managing stakeholder expectations and appropriate reporting.

Operating System Review
A review of the Operating System configuration needs to look at
a) The number of file descriptors available and whether that matches the value recommended for Weblogic
b) Various TCP settings (also called NDD parameters) which will affect how long it takes to recycle a closed or dropped connection.
2 top settings are:
• File Descriptor limit: increase to 8192 if not 65536. A detailed follow-up on File Descriptors is published here.
• tcp_time_wait_interval: change to 60000 [The default in Solaris 8 is 4 minutes used to keep a socket open in TIME_WAIT state, even after the response is provided to the client, set this down to 1 minute. The default in Solaris 9 is 1 minute]
Check the Oracle site for the latest recommended values

Database Usage and JDBC Drivers
a) If you get Out Of Memory errors occurring in JDBC calls, it is recommended that the JDBC driver be upgraded to the latest version.

b) Prepared Statement caching is a feature that allows prepared statements to be held in a cache on the database so they do not have to be re-parsed for each call. This functionality needs to be enabled per Connection Pool and can have a significant impact on the performance of the pools. But this needs to be validated with a focussed round of Performance testing.

It should be noted that for every Prepared Statement that is held in cache, a cursor is held open on the database for each connection in the pool. So if a cache size of 10 is used on the abcPool, and the pool has a size of 50 then 500 open cursors will be required. Repeatable load tests will highlight any gains achieved by enabling this caching.

Review JDBC Pool Sizes
Review connection pool versus the number of Execute threads. Usually keep Pool size close to Execute thread size. Note: This applies to versions earlier than Weblogic 9. See detailed explanation below.

If the JDBC pool size is quite less compared to the Thread size, there is the potential to negatively impact performance quite dramatically, since threads have to wait for connections to be returned to the pool.
Your most frequently used pool should have their minimum (initial) and maximum sizes increased to the number of Execute threads plus one. This will mean there is an available connection for every thread.
One comment on pool sizing it is beneficial where ever possible to have the initial and max connections set to the same size for a JDBCPool as this avoids expanding/shrinking work that can be costly, for both the establishment of new connections while expanding the pool and housekeeping work for the pool.

However it is also recommended to monitor the JDBC pools during peak hours and see how many connections are being used at maximum. If you are not hitting the MaxCapacity, it is useful to reduce the MaxCapacity to avoid unnecessary open cursors on the database.

Note: As of Weblogic 9 and higher, Execute Queues are now replaced by Work Managers. Work Managers can be used for JDBC pools by defining the max-threads-constraint to define how many threads to allocate for a particular Datasource.
It is possible to run Weblogic 9 and 10 with the Execute Queues as available earlier. This is not recommended since Work Managers are self-tuning and more advanced than Execute Threads.

WebLogic Server, Version 8.1, implemented Execute Queues to handle thread management in which you created thread-pools to determine how workload was handled. WebLogic Server still
provides Execute Queues for backward compatibility, primarily to facilitate application
migration. However, when developing new applications, you should use Work Managers to perform thread management more efficiently.
You can enable Execute Queues in the following ways:
􀁺 Using the command line option
-Dweblogic.Use81StyleExecuteQueues=true
􀁺 Setting the Use81StyleExecuteQueues property via the Kernel MBean in config.xml.
Enabling Execute Queues disables all Work Manager configuration and thread self tuning.
Execute Queues behave exactly as they did in WebLogic Server 8.1.

Database Persistent JMS Queues

Verify whether the database architecture is such that persistent JMS queues use the same database instance as a message store as your Weblogic portal uses for data.
As the volumes on these queues increase this could significantly degrade the performance of the portal by competing for valuable CPU cycles on the database server.

1. Move the message store for persistent queues to a separate database instance from that used by most of the JDBC pools belonging to the Weblogic server. This will prevent increases in message volumes from adversely affecting the performance of the database, which would also slow the portal applications down and vice-versa.

2. Implement paging with a file-store. This allows the amount of memory consumed by JMS queues to be restricted by paging message contents to disk and only holding headers in memory. Note that this does not provide failover protection in the way persistence does and performs better with a paging file-store than a paging JDBC store.

3. It is recommended that a review is also undertaken to determine exactly which queues are persisted and whether they truly need to be. The performance gains from switching to non-persisted queues are substantial, and guaranteed delivery is not always required.

Review Number of Execute Threads
A common mistake made by Support teams when seeing Stuck threads is to increase the number of execute threads in a single ‘default’ queue. At one time, I have worked in a project which ran the Weblogic server with 95 threads.

This figure is very high and results in a large amount of context-switching as load increases, which consumes valuable CPU cycles. Because threads consume memory, you can degrade performance by increasing the value of the Thread Count attribute unnecessarily.

Taking a Thread Dump when the server is not responding will show what the threads are doing, and help identify whether there is an application coding issue or deadlock occuring. Use Samurai to analyze these as earlier posted
It is recommended that regular monitoring of the number of idle threads and the length of queued requests for each execute queue is set up via MBeans. This allows the Support teams to plot a graph of utilization and validation of any changed values.
Note: As of Weblogic 9 and higher, Execute Queues are now replaced by Work Managers. Another good link is here

Use Dedicated Execute Thread Queue for Intensive applications

As the number of threads is small, and if a particular application is seen to utilize a majority of the execute, the following 2 approaches are suggested to resolve the issue:
1. Review the design of the offending application to determine whether it really needs so many threads.
2. Move the offending application to a dedicated execute queue, with enough threads allocated to this queue. This will prevent it from starving the main server of threads and allow the ‘default’ queue to remain with a lower number of Execute threads. This split of Execute Queue can be done at servlet or webapp level. Mail me if you need an example, we've done both successfully in WL 8.1 and 9.

Note: However, as of Weblogic 9 and higher, Execute Queues are now replaced by Work Managers. You can use a Work Manager to dedicate resources at Application, Web App, EJB level.

Review Java VM Settings
Tuning the JVM settings for the Total Heap, and Young/Old Generations is essential to regulate the frequency of Garbage Collection on the servers. The basic primer is on the Sun website, and a follow up of actual values and learnings is published here. The most essential ones are Xms, Xmx for the total Heap and NewSize, NewRatio for the Young Generation. Also set PermSize and MaxPermSize appropriately to avoid consuming high memory

Other Areas

1. To speed up server start times, do not delete the .wlnotdelete directories at startup - Unless you are deploying changed application jars and code.
Be aware you might occasionally see a problem shutting down the server which goes into an UNKNOWN state due to too many old temp files and wl_internal files.
You will get the dreaded error below which can only be resolved by killing the process and clearing out all temp files, .lck files etc within the domain. The files are under DOMAIN_HOME/servers//



weblogic.management.NoAccessRuntimeException: Access not allowed for subject: principals=[weblogic, Deployers], on ResourceType: ServerLifeCycleRuntime Action: execute, Target: shutdown
        at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:195)
        at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:224)

2. Avoid the URLEncoder.encode method call as much as possible. This calls is not optimal in most JDKs < 1.5 and is often found as memory and CPU hotspot issue.

3. Check the network connection between WebLogic and the database. If Thread dumps show that threads are often in a suspended state (waiting for so long that they were suspended) while doing a socket read from the database.
The DBA wouldn't see this as a long-running SQL statement. This needs to be checked out at the network level.

4. Switch off all DEBUG options on Production on app server as well as web server and web server plugins.

5. Ensure log files are rotated so that they can be backed up and moved off the main log directory. Define a rotation policy based on file size or fixed time (like 24 hours)
However also note that: On certain platforms, if some application is tailing
the log at the time of rotation, the rotation fails. Stop the application tailing and reopen the tail after the rotation is complete.

6. Do not use "Emulate Two-Phase Commit for non-XA Driver" for DataSources.
It is not a good idea to use emulated XA. It can result in loss of data integrity and can cause heuristic exceptions. This option is only meant for use with databases for which there is no XA driver so that the datasources for these pools can still participate in XA transactions.
If an XA driver is available (and there is for Oracle), it should be used. If this option is selected because of problems with Oracle's Thin XA driver, try the newest version, or pick a different XA driver.

7. The <save-sessions-enabled> element in web.xml controls whether session data is cleaned up during redeploy or undeploy.
It affects memory and replicated sessions. The default is false. Setting the value to true means session data is saved and this is an overhead.

8. If firewalls are present between Weblogic server and the database or an external system connecting via JMS, this can cause transactional and session timeout issues. The session timeout on the firewall shoudl be configured to allow for normal transaction times.
In the case of JMS, transactional interoperation between the two servers can be compromised and hence it is beneficial to open the firewall between the two servers so that RMI/T3 connections can be made freely.

Any queries or clarifications, leave me a comment and I'll try to get back.

Slow Weblogic response - JSP and Servlet Reload

2009-05-06T01:37:00.000-07:00

The top Weblogic tuning tip I've used over the last few years is the one below. This article deals with Weblogic 8 and 9.

You know the symptoms:

JEE Application works okay on Unit Integration and Test servers.
But once you're into Performance and Stress Testing - it's totally choked up - pages are returned so slowly - Project Manager is asking for a code review to find the blockers - etc.

The first thing to do on a slow JEE application is to take a thread dump, see more on that here

Now, the thing to look for in the Thread Dump, is multiple threads doing this:


weblogic.servlet.jsp.JspStub.checkForReload(JspStub.java:144)

or this:


waiting for monitor entry [c4f7f000..c4f7fc28]
     at weblogic.servlet.internal.ServletStubImpl.checkForReload(ServletStubImpl.java:771)

This is not a stuck thread within your code – but this is a weblogic.xml setting for checking each time if your servlet class or JSP has been changed.

As documented here:

http://e-docs.bea.com/wls/docs92/webapp/weblogic_xml.html#wp1038491

Sets the interval, in seconds, at which WebLogic Server checks to see if JSP files have changed and need recompiling. Dependencies are also checked and recursively reloaded if changed.

If set to 0, pages are checked on every request. If set to -1, page checking and recompiling is disabled.

Most users set this to -1 to disable in Production unless they're altering jsps on the fly (which is not normal practice on a production system).

The correct syntax for this is:

<weblogic-web-app>
<jsp-descriptor>
<jsp-param>
<param-name>pageCheckSeconds</param-name>
<param-value>-1</param-value>
</jsp-param>
</jsp-descriptor>

The similar value for servlets is set using the servlet-reload-check-secs within the tags below.

<container-descriptor>
<servlet-reload-check-secs>-1</servlet-reload-check-secs>
</container-descriptor>
</weblogic-web-app>

On non-production environments, you can set this to a reasonable value such as 600 (in seconds, which is 10 minutes) to allow for changed JSPs you drop in/FTP to the deployed environment, and want those to reflect sooner.

The other thing we must do on a production system is to precompile the JSPs as part of the build, and before deploying to Live.

Else, even the first time JSP compilation is quite slow and this must be avoided on the Production system.

For the same solution on Weblogic 10, look at an updated article.

com.vladium.emma.EMMARuntimeException: [CLASS_STAMP_MISMATCH]

2009-04-15T02:28:00.000-07:00

Emma is used for finding the code coverage of JUnit test classes.

When running Emma with a JEE container, the source .class files (not the .java files) are first instrumented and then deployed on the J2EE server.

Then the test classes are executed from ant command line or via an Eclipse plug-in, and this will write out the coverage data into .em or .ec files.

Sometimes, an error observed when running Emma is


emma.report:
   [report] processing input files ...
   [report] 2 file(s) read and merged in 172 ms
   [report] com.vladium.emma.EMMARuntimeException: [CLASS_STAMP_MISMATCH] runtime version of class [com.xyz.action.MyAction] in the coverage data is not consistent with the version of this class in the metadata, possibly because stale metadata is being used for report generation.

   [report]     at com.vladium.emma.report.ReportDataModel.getView(ReportDataModel.java:95)
   [report]     at com.vladium.emma.report.AbstractReportGenerator.initialize(AbstractReportGenerator.java:207)
   [report]     at com.vladium.emma.report.html.ReportGenerator.process(ReportGenerator.java:85)
   [report]     at com.vladium.emma.report.ReportProcessor._run(ReportProcessor.java:254)

To calculate the coverage, EMMA combines data of two types:
metadata (static info about your Java classes, methods, lines, and basic blocks) and runtime coverage data (which basic blocks have been executed)

The error occurs when there is a mismatch in the versions of the classes used while recording the metadata.em and coverage.em files. The runtime version of the class in the coverage data is not consistent with the same class in the metadata

In simpler terms, the in-container application jars deployed on the JEE server are not in sync with the currently instrumented source from where the emma is being run, hence the class stamp difference in .emma files – metadata.em and coverage.em

Update -
A couple of comments posted below have some more checks and solutions to this problem.

Running Maven offline using local Repository

2009-04-02T03:11:00.001-07:00

Sometimes you see Maven downloading POM files for dependencies which are already in your local repository and you're wondering why it does this.


Downloading: http://repo1.maven.org/maven2//org/apache/xmlgraphics/fop/0.95/fop-
0.95.pom
Downloading: http://repo1.maven.org/maven2//org/apache/xmlgraphics/xmlgraphics-c
ommons/1.3.1/xmlgraphics-commons-1.3.1.pom
Downloading: http://repo1.maven.org/maven2//org/apache/xmlgraphics/batik-svg-dom
/1.7/batik-svg-dom-1.7.pom

The reason it does is because when these were installed in the local repository, it was not given a flag of -Dgenerate.pom=true

If you run the install with this flag, then it will generate and install a local POM file for you, and will not download the POM from the internet repository.

An example below


mvn install:install-file -DgroupId=%GROUP_ID% -DartifactId=%ARTIFACT_ID% -Dversion=%VERSION% -Dfile=%COMPONENT%.jar -Dpackaging=jar -DgeneratePom=true

There are 2 other ways to run the build without connecting to the internet or network repository each time - provided you have all the required dependencies available offline.

1. Run the maven build in offline mode


mvn -o install

[INFO]
NOTE: Maven is executing in offline mode. Any artifacts not already in your loca
l
repository will be inaccessible.

...

2. Point the repository to the local file system.

On my local build, if running repeated code builds when I know there is no change in dependency, I have a profile set up to point the repository to a local file system.

In the repository definition, instead of
<url>http://repo1.maven.org/maven2/</url>

use the file:// path

<repository>
<id>central</id>
<url>file://D:\mavenrepo</url>
</repository>

You will still see the message indicating downloading of POM, but this time from the local file system.
And this will be much faster.


Downloading: file://D:\mavenrepo\cactus\1.7.2\cactus-1.7.2.pom
Downloading: file://D:\mavenrepo\strutstest\2.1.3\strutstest-2.1.3.pom

java.lang.ClassNotFoundException: org.apache.tools.ant.Main

2009-03-13T01:55:00.000-07:00

This error should not be occuring when you install Ant out-of-the-box and run it. However, on the rare occasions it does occur - the solution is quite simple.



java.lang.ClassNotFoundException: org.apache.tools.ant.Main
        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
        at org.apache.tools.ant.launch.Launcher.run(Launcher.java:244)
        at org.apache.tools.ant.launch.Launcher.main(Launcher.java:67)

Take a look at the ANT_HOME/lib folder - and ensure that the ant.jar is in there and showing a size of 1MB or higher.

It is usually due to a corrupted version of ant.jar in the above path.

Executing multiple mvn commands from a Windows bat file

2009-03-09T23:51:00.000-07:00

When running Maven, if you try to run multiple Maven commands in a batch (.bat) file, it only runs the first one and exits to the command prompt


mvn clean
mvn package

This occurs since mvn itself is a bat file.

To overcome this, you need to stop Windows passing control to the mvn.bat script by using the call command for each call to mvn.


call mvn clean
call mvn package

Additionally, to catch failures you will need to do this


call mvn clean
echo Exit Code = %ERRORLEVEL%
if not "%ERRORLEVEL%" == "0" exit /b

call mvn package
echo Exit Code = %ERRORLEVEL%
if not "%ERRORLEVEL%" == "0" exit /b

A failure throws an ERRORLEVEL higher than 0.
Otherwise it will keep moving ahead even if it fails.

Note: if you're using maven 2.0.7 or older, there is a possibility that the mvn does not return an ERRORLEVEL > 0 for the process to exit.

Lots of examples are available here http://www.google.co.uk/search?hl=en&q=mvn+exit+ERRORLEVEL&meta=

A simple test to check if your version allows this - run the commands below in a bat file.


call mvn unknown
echo Exit Code = %ERRORLEVEL%

You should get a Failure with exit code 1 as below.
If it does not fail, then upgrade to a higher version of Maven in which the issue is fixed.


D:\maven-2.0.7\bin>call mvn unknown
[INFO] Scanning for projects...
[INFO] ----------------
[ERROR] BUILD FAILURE
[INFO] ----------------
[INFO] Invalid task 'unknown': you must specify a valid lifecycle phase, or a goal in the format plugin:goal or pluginGroupId:pluginArtifactId:pluginVersion:goal[INFO] ----------------
[INFO] For more information, run Maven with the -e switch
[INFO] ----------------
[INFO] Total time: < 1 second
[INFO] Finished at: Tue Mar 10 12:07:12 GMT+05:30 2009
[INFO] Final Memory: 1M/2M
[INFO] ----------------
Exit Code = 1

Update

Learnt the hard way that the call to mvn must immediately be followed by the checking of the ERRORLEVEL.

Our build.bat script measures time taken for each step like below:


time /t
call mvn install
time /t
if not "%ERRORLEVEL%" == "0" exit /b

In this case, it loses the ERRORLEVEL value from the mvn command.
Hence we need to ensure:


time /t
call mvn install
if not "%ERRORLEVEL%" == "0" exit /b
time /t

A Tribute to the Divine Genius of A R Rahman

2009-02-23T22:13:00.000-08:00

Having won at the Oscars and the Golden Globes, A. R. Rahman is getting the recognition he deserves for his genius. Personally I do not rate Slumdog Millionaire among his top 5 albums even. He's composed for over 110 albums and a good debate on his best work is available here

My post today is on Rahman's truly divine gift. Some of his music makes one feel blessed to be just listening to the soul-stirring melody - and truly truly you close your eyes and feel you are in another land - sitting obeisant in front of God.
His own philosphy as written on this blog states
"Music is my means of connecting with the divine. What cannot be put into words, can be expressed through music."

It's well known that he is a follower of Sufism - and there are various songs which evoke that music. But you sometimes cannot believe that one man can put together all of this music. A true disciple of music in it's forms and variations, his music has blended such diverse techniques to create sound you never could imagine.
Among all the musical wonders he's produced over the ages the ones I'm writing about today are the devotional tracks appearing in various OSTs.

Sufi / Qawwali

“Khwaja Mere Khwaja” from Jodhaa Akbar

"Arziyan" from Delhi-6. The first two lines of this humble tribute sung by Javed Ali and Kailas Kher are just lovely and the rest is very well arranged.

"Tere Bina" from Guru

The bhajans

“O Paalanhaare” from Lagaan (sung by Lata Mangeshkar and Udit Narayan).

"Mann Mohana” from Jodhaa Akbar (sung by Bela Shende) - the violin interlude is such splendour. This is my favourite Rahman song. It must be heard with one's eyes closed to really feel the maetro's magic and oneness with God. Note to self: dont listen while driving.

"Aarti (Tumre Bhavan Mein)" from Delhi-6 is extremely simple with just the sitar stringing in the background but so moving. Sung in unison by Rekha Bharadwaj, Kishori Gowarikar, Shraddha Pandit and Sujata Majumdar

And his interpretation of the Sikh prayer "Ik Onkar" from Rang De Basanti - also picturized very well in the movie - simple and devotional.

Also must mention "Maa Tujhe Salaam" part of an album released to celebrate India’s 50th year of Independence in 1997.

Bad Java code with good intentions

2009-02-20T01:38:00.000-08:00

My friend Deepak sent me a real-life example of the following Java code. Let's give the developer the credit of following defensive coding techniques. [Rather than calling him a dumb-a$$]


String flashPath = (String)request.getAttribute("flashPath");
String flashPathPath = flashPath.toString();

Sathya improved on it by adding checks


String flashPath = (String)request.getAttribute("flashPath");
If (flashPath instance of java.lang.String){
    String flashPathPath = flashPath.toString();
}

So let's work on this some more



String flashPath = (String)request.getAttribute("flashPath");
If (flashPath instance of java.lang.String){
            try{
               String flashPathPath = (String) flashPath.toString(); //Let's cast it once more to be sure, if it escaped the first time, it'll surely get caught here.
            }
            catch (ClassCastException e)
            {
               // Don’t cast it 
              String flashPathPath = flashPath.toString(); 
            }
}

Now this is open to all to further obfuscate.
How complicated can you make one line of code?

Now Playing Jan/Feb 2009

2009-02-13T03:56:00.000-08:00

Music albums that I've picked up and liked over the last 2 months:

Best of Led Zeppelin
Keane - Under the Iron Sea
The Weepies - Say I Am You
Joshua Radin - Simple Times
Delhi 6 - OST
Luck By Chance - OST
Dashboard Confessional - Dusk and Summer
Dev D - OST

I seem to like most of the Indie picks from Zach Braff's movies - "The Last Kiss" and "Garden State" - and the Scrubs OST.
In addition to the above albums, Cary Brothers, Zero 7 and Imogen Heap were stumbled upon off various Scrubs episodes.

junit.report fails with OutOfMemoryError

2009-02-12T00:59:00.000-08:00

Running too many testsuites in junit.report causes a failure with the message

2009-02-11 08:33:25,664 [Thread-15] INFO ScriptRunner - junit.report:
2009-02-11 08:33:25,667 [Thread-15] INFO ScriptRunner - [mkdir] Created dir: /a01/home/Construction_temp/reports/ebilling/junit
2009-02-11 08:33:41,604 [Thread-14] WARN ScriptRunner - java.lang.OutOfMemoryError: PermGen space
2009-02-11 08:33:41,605 [Thread-14] WARN ScriptRunner - PermGen space

Resolution:

in the build.bat or build.sh, add more memory to the JVM process like so:

ANT_OPTS="-Xms128m -Xmx512m -XX:MaxPermSize=256m"
export ANT_OPTS=$ANT_OPTS

Useful Unix Commands - Part 2

2009-02-05T22:03:00.000-08:00

A selection of useful find commands

=== Find User ===

fuser displays the PIDs of processes using the specified files or file systems.
fuser command is very useful in getting rid of the .nfs files created when processes are killed. The .nfs* names refer to files that have been deleted but are still being held open by a running process.

You can find .nfs files using ls -la

rm does not work in removing .nfs files - use fuser to find the process that's locking the file and then kill the process

 /usr/sbin/fuser {filename}

 kill -9 {pid}

If you stop the processes that have them open, the files will be tidied-up and removed by the file server.

If you're interested a bit of general background follows:

Most operating systems, including UNIX, operate a policy of not actually removing a deleted file (and freeing up it's data blocks) until the last process that has the file open closes it. So, if a running process has a file open and you use the rm(1) command to delete the file the data blocks on the disk will not be freed up by the OS until the process that has the file open closes it.

On a UNIX host using local disk store this behaviour can manifest itself it some seemingly confusing situations. For example, you may wish to free up some space on a file system that's used 900 MB of it's 1GB quota. You have a large file, 200MB say, named myjava.jar that you believe is no longer required, but is actually currently open in your WebLogic server. Not knowing this you delete myjava.jar and do an ls(1) command to see that the file is no longer listed. However, when you use the df(1) command it still reports that 900 MB of it's 1GB quota is used because your WebLogic server still has the file open. When you shutdown the WebLogic server or the server closes the file the disk space will be released and df(1) will report 700MB of it's 1GB is used.

If the file that is removed is on NFS mounted store then it is possible for a file to be deleted on one client whilst still being open on another client. In this situation the same rule of not actually deleting the file until the last process with it open closes it still applies. However, in order for the NFS file handle used by the client that still has the file open not to be broken a filename reference must be maintained. In order to achieve this and remove the files name from the directory ( so it doesn't show up in an ls command output) the NFS file server renames the deleted file to a name beginning '.nfs'. These are the files you are seeing. When the last process with these files open dies or closes them they will be tidied up and removed. Trying to delete them before they are closed will only result in the file being renamed again.

=== Find Class within Jar file ===

To find a class within a binary jar file

 for i in `ls *.jar`; do (jar tf $i  grep '{classname}' ) && echo $i;done

If you want to search within all subdirectories use this

 for i in `find . -name ‘*.jar’`; do (jar tf $i  grep '{CLASSNAME}' ) && echo $i;done

=== Find string within file of name ===


find . -name *.xml -exec grep  {} \;

example

find . -name web.xml -exec grep -i servlet {} \;

=== Grep within zip file without unzipping it ===


gzip -c -d {file}.gz | grep {string}

Example

gzip -c -d admin_access.log0001_4Dec08_0147.gz | grep -i adq | wc -l

=== Find process using the port ===

The easy way to do this is using netstat passing the port number

netstat -a | grep 61014

If that does not help getting a pid, run this line below


for i in `ls /proc`; do pfiles $i | grep AF_INET | grep 61014 ; done

Output:

pfiles: permission denied: 12363
sockname: AF_INET 0.0.0.0 port: 61014
pfiles: permission denied: 12384

this shows the process appearing in between pids 12363 and 12384 uses that port.

ls /proc
gives the PIDs in order as .. 12363, 12369, 12384 ..

ps -ef grep 12369
wlsuser 12369 7852 0 01:07:40 ? 4:42 /opt/bea/jdk142_05/bin/java -server -DresKBD -Xms512m -Xmx512m -XX:MaxPermSize=

gave the process details which was using the port

Useful Unix commands - part 1

2009-02-03T22:29:00.000-08:00

CPU Usage
The prstat command displays information about active processes on the system and resource consumption. By default, prstat displays information about all processes sorted by CPU usage.

The prstat -s cpu -n 5 command is used to list the five processes that are consuming the most CPU resources. The -s cpu flag tells prstat to sort the output by CPU usage (This is a default flag set, so can be skipped). The -n 5 flag tells prstat to restrict the output to the top five processes.

Adding the -a option to any prstat command will identify how many processes each user is using, what percent of the CPUs, and how much memory, they are using on a system

prstat -a

The ps command displays information about currently running processes. Pipe the output to search for specific processes

 ps -ef  | grep java

/usr/ucb/ps -auxwww | grep {pid}

This command gives the complete classpath and start parameters of the process mentioned in pid

Example:

/usr/ucb/ps -auxwww | grep 19138

user 19138 0.4 2.31272184732136 ? S Jun 01 342:11 /resolve/j2sdk1.4.2_03/bin/java -Xmx1024m -Xms1024m -XX:NewSize=341m -XX:MaxNewSize=341m -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:+DisableExplicitGC -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -Xloggc:/resolve/CPEE3112/composer/log/pa3.gc -Drmiport=61050 -Dg2config.home=/resolve/CPEE3112/composer/conf -Dg2config.baseipport=61020 -Dg2config.instance=Engine14 -Dg2.processid=3 -cp /resolve/lib/any_custom_libs.jar:/resolve/lib/G2.jar:/resolve/lib/oracle_10_2_0_1_0jdbc.jar com.xyz.platform.Main -recover &


prstat -L -p {pid}

The above cmd gives the summary of lightweight processes which make up the process. More is at an example I gave here

The top command provides an overview of CPU and memory utilization, and a list of the top consumers of CPU, it updates it's display every few seconds so you can monitor continuously. Use http://www.groupsys.com/top/display/ to view details of the stats shown


top

sar -s

Memory Usage
vmstat 5 vmstat reports virtual memory statistics regarding kernel, thread, virtual memory, and disk, trap, and CPU activity. vmstat also helps calculate the average page scan rate.

Below is the vmstat output


vmstat 5



kthr memory page disk faults cpu

r b w swap free re mf pi p fr de sr s0 s1 s2 s3 in sy cs us sy id

0 0 0 11456 4120 1 41 19 1 3 0 2 0 4 0 0 48 112 130 4 14 82

0 0 1 10132 4280 0 4 44 0 0 0 0 0 23 0 0 211 230 144 3 35 62

0 0 1 10132 4616 0 0 20 0 0 0 0 0 19 0 0 150 172 146 3 33 64

0 0 1 10132 5292 0 0 9 0 0 0 0 0 21 0 0 165 105 130 1 21 78

Note: The first line of vmstat shows a cumulative value and must be ignored. sr is the pages scanned by clock algorithm.

On Unix, to find out how much RAM is present on a machine, use


prtconf grep 'Memory size:'

On Linux, the same is done using


free -m

Disk Space


df -ek

du -h

Findbugs XSLT style Error java.lang.reflect.InvocationTargetException

2009-01-29T21:52:00.000-08:00

Tried running out-of-the-box Findbugs 1.3 with Ant 1.6 and Ant 1.7, on JDK 1.4 and JDK 1.5
It throws an error for the task findbugs.report which is a simple XSLT transform using a stylesheet. Peculiarly this fails only on Windows but works fine on Unix.
build.xml contains:

<style
basedir="${build.report.temp}"
destdir="${build.report}/findbugs"
includes="findbugs-post.xml"
style="${script.tools}/findbugs/tools/xsl/report.xsl">
<param
name="messages.xml"
expression="${script.tools}/findbugs/tools/xsl/messages.xml"/>
<param
name="output.dir"
expression="${build.report}/findbugs"/>
</style>

The error is


[style] Processing D:\Construction\reports\ebilling\tmp\findbugs-post.xml to D:\Construction\reports\ebilling\findbugs\findbugs-post.html 
[style] Loading stylesheet D:\Construction\buildtools\bin\findbugs\tools\xsl\report.xsl 
[style] : Error! java.lang.reflect.InvocationTargetException 
[style] Failed to process D:\Construction\reports\ebilling\tmp\findbugs-post.xml
BUILD FAILED


D:\Construction\buildtools\dbscripts\findbugs.xml:67: javax.xml.transform.TransformerException: com.sun.org.apache.xalan.internal.xsltc.TransletException: java.lang.reflect.InvocationTargetException

The problem is the <param> passed in with an absolute path fails for some reason - but it works when the path is relative to the stylesheet referenced in the style attribute. In this case the stylesheet is at style="${script.tools}/findbugs/tools/xsl/report.xsl"
So we changed:

<param expression="${script.tools}/findbugs/tools/xsl/messages.xml" name="messages.xml">

to a path relative to the report.xsl

<param expression="./messages.xml" name="messages.xml">
and that worked.

App Server High CPU or Low Threads - Support Team Actions

2009-01-13T01:58:00.001-08:00

Occasionally Weblogic/JEE app servers run into High CPU (>70% or as per your app-defined thresholds) , then below is a list of actions and investigations that the support teams need to do.

Check prstat –L –p Capture into a file if required.

Output kill -3 into a file i.e. take thread dump. On Weblogic 10,you can do this from the Admin console as well

Output pstack into a file

Repeat this 4 times at 5 second intervals. At the end, you will have 5 sets of prstat, TD and pstack files.

Run the TDs through Samurai or TDA to identify deadlocks. These tools show a graphical view of what each thread is doing at each point of time (when the thread dump was taken).
The image below from Samurai shows RED blocks which are Stuck, and Green with < in them. Both of these need to be analyzed by clicking on the link in blue and seeing what line the thread is at. The Green ones with < might not be stuck, but they were at the same line as in previous thread dump and that indicates a potential problem.
Then analyze the prstat and pstack for any particular thread issues.

On the prstat, look for the LWP number on the right hand corner for high cpu and any long time in action.

Convert the lwp number to hexa decimal value as follows:
echo nawk '{printf("%d=0x%x\n",$1,$1)}'

In the thread dump, search for "nid=" these threads could be the issue for the application.

In the pstack outputs, check for the LWP numbers and confirm that the threads show the java trace.

Below is an example - Note this is indicative and not a proper instance of a slow-performing server.

1. prstat -a shows the java process consuming 42.6% and it has 180 LWP (Light-weight processes) running.


PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP

20677 wlsuser 1116M 526M sleep 1 0 2:20:54 42.6% java/180
11922 wlsuser 232M 110M sleep 1 0 3:41:59 0.1% java/157
29861 noaccess 198M 72M sleep 8 0 32:25:49 0.0% java/55
18133 root 630M 55M sleep 27 0 12:25:57 0.0% java/60
12107 wlsuser 927M 288M sleep 1 0 3:51:28 0.0% java/187
29654 wlsuser 1270M 206M sleep 1 0 0:54:26 0.0% java/158
2633 wlsuser 899M 217M sleep 1 0 7:06:10 0.0% java/382
7431 wlsuser 880M 253M sleep 5 0 1:44:24 0.0% java/186
1058 wlsuser 881M 241M sleep 1 0 3:11:35 0.0% java/182
9864 wlsuser 4336K 3952K cpu21 59 0 0:00:00 0.0% prstat/1
13789 wlsuser 1090M 208M sleep 17 0 0:30:44 0.0% java/210
13147 wlsuser 635M 168M sleep 15 0 7:17:05 0.0% java/180
7724 root 5728K 4112K sleep 60 0 33:54:37 0.0% vasd/1
18131 root 2272K 1088K sleep 59 0 8:18:11 0.0% wrapper/1
15397 root 2272K 1296K sleep 59 0 3:33:59 0.0% procmon/1
2. prstat -Lp 20677 gives the breakdown of these 180 LWPs
The top 2 can be seen to be consuming 10% CPU each and are running for longer than 15 minutes.


PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/LWPID

20677 wlsuser  1116M  526M sleep   58    0   0:16:03 10.3% java/120

20677 wlsuser  1116M  526M sleep   24    0   0:15:15 10.1% java/123

20677 wlsuser  1116M  526M sleep    3    0   0:03:36 4.0% java/23

20677 wlsuser  1116M  526M sleep    5    0   0:01:42 2.0% java/21

20677 wlsuser  1116M  526M sleep    9    0   0:03:38 2.0% java/20

20677 wlsuser  1116M  526M sleep    9    0   0:00:38 1.7% java/19

20677 wlsuser  1116M  526M sleep    4    0   0:00:42 0.0% java/18

20677 wlsuser  1116M  526M sleep    4    0   0:00:36 0.0% java/17

20677 wlsuser  1116M  526M sleep    4    0   0:00:34 0.0% java/16

20677 wlsuser  1116M  526M sleep    1    0   0:00:42 0.0% java/15

20677 wlsuser  1116M  526M sleep    4    0   0:00:42 0.0% java/10

20677 wlsuser  1116M  526M sleep    6    0   0:07:43 0.0% java/8

20677 wlsuser  1116M  526M sleep    3    0   0:00:39 0.0% java/7

20677 wlsuser  1116M  526M sleep    1    0   0:00:40 0.0% java/4

20677 wlsuser  1116M  526M sleep    8    0   0:00:39 0.0% java/2

20677 wlsuser  1116M  526M sleep   29    0   0:04:55 0.0% java/121

20677 wlsuser  1116M  526M sleep    1    0   0:11:52 0.0% java/126

20677 wlsuser  1116M  526M sleep    1    0   0:00:05 0.0% java/119

20677 wlsuser  1116M  526M sleep   43    0   0:00:27 0.0% java/122

20677 wlsuser  1116M  526M sleep   28    0   0:00:03 0.0% java/118

20677 wlsuser  1116M  526M sleep    1    0   0:12:43 0.0% java/125

3. Run the cmd to convert the LWP decimal to Hex
echo 120 nawk '{printf("%d=0x%x\n",$1,$1)}' 120=0x78
echo 123 nawk '{printf("%d=0x%x\n",$1,$1)}' 123=0x7b
Now we need to investigate what these 2 threads are doing.

4. Check the Thread dump for the nid=0x78. It shows


"ExecuteThread: '83' for queue: 'wlr3Queue'" daemon prio=5 tid=0x00a886f0 nid=0x78 waiting for monitor entry [c3afe000..c3affc28]

at org.apache.log4j.Category.callAppenders(Category.java:204)

- waiting to lock <0xdc8e9080> (a org.apache.log4j.spi.RootLogger)

at org.apache.log4j.Category.forcedLog(Category.java:391)

at org.apache.log4j.Category.debug(Category.java:260)

...

5. Running the above TD through Samurai also shows that Execute Thread 83, has been stuck for a while. It also shows that the same object <0xdc8e9080> is being locked by another Thread which happens to be the nid=0x7b


"ExecuteThread: '86' for queue: 'wlr3Queue'" daemon prio=5 tid=0x00a85730 nid=0x7b runnable [c3efd000..c3effc28]

at java.io.FileOutputStream.writeBytes(Native Method)

at java.io.FileOutputStream.write(FileOutputStream.java:260)

at java.io.BufferedOutputStream.write(BufferedOutputStream.java:106)

- locked <0xd8b55a70> (a java.io.BufferedOutputStream)

at java.io.PrintStream.write(PrintStream.java:258)

- locked <0xd88512c8> (a java.io.PrintStream)

at sun.nio.cs.StreamEncoder$CharsetSE.writeBytes(StreamEncoder.java:336)

at sun.nio.cs.StreamEncoder$CharsetSE.implFlushBuffer(StreamEncoder.java:404)

at sun.nio.cs.StreamEncoder$CharsetSE.implFlush(StreamEncoder.java:408)

at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:152)

- locked <0xdce03ae8> (a java.io.OutputStreamWriter)

at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:213)

at org.apache.log4j.helpers.QuietWriter.flush(QuietWriter.java:58)

at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:316)

at org.apache.log4j.WriterAppender.append(WriterAppender.java:160)

at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)

- locked <0xdc8eeb60> (a org.apache.log4j.ConsoleAppender)

at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)

at org.apache.log4j.Category.callAppenders(Category.java:206)

- locked <0xdc8e9080> (a org.apache.log4j.spi.RootLogger)

6. Thus there is an issue around the log4j writing into log files, which needs further investigation.

In this case, one issue identified was that the log4j was in DEBUG mode which should not be the case in a Production platform. Thus there was unnecessary logging taking place.

7. Finally, a look at the pstack outputs for the two LWP 120 and 123 confirms what the LWPs were doing.

----------------- lwp# 120 / thread# 120 --------------------
ff341758 lwp_mutex_timedlock (f66e68, 0)
fedbbb60 __1cLOptoRuntimebAcomplete_monitor_locking_C6FpnHoopDesc_pnJBasicLock_pnKJavaThread__v_ (da5a6848, c3aff0b0, a886f0, da5b6980, d6d15e88, 0) + 84
f9c342dc ???????? (da5a6848, c3aff0b0, da5b6980, d6d15e70, 0, c3aff0f8)
fa643500 ???????? (da5c9fc0, 0, da5c9fc0, da5b6980, db3b3af8, 0)
fa5c0e2c ???????? (da5c9fc0, f1c8e7a0, da5b6980, d6d15e70, 0, c3aff0f8)
fa762524 ???????? (da5b65f8, d6d15e70, 0, f9c15ea0, 8, c3aff178)
f9c4eec4 ???????? (da5b65f8, f52b9bf0, f0ab1c94, f9c160d0, 8, c3aff0f8)
f9c05a8c ???????? (c3aff29c, b6, 0, f9c15ea0, 8, c3aff178)
f9c05804 ???????? (c3aff33c, b6, 0, f9c15e50, c, c3aff210)
f9c05804 ???????? (c3aff3cc, b6, 0, f9c15e50, c, c3aff2d8)
f9c05804 ???????? (c3aff464, b6, 0, f9c15e50, c, c3aff358)
f9c05804 ???????? (c3aff4f4, f1a8a5c8, 0, f9c15e50, c, c3aff400)
f9c05a8c ???????? (c3aff58c, f084f340, 0, f9c163d0, c, c3aff490)
f9c059d8 ???????? (c3aff674, b6, 0, f9c163d0, 4, c3aff518)
f9c05804 ???????? (c3aff6fc, b6, 0, f9c15e50, 10, c3aff608)
f9c05804 ???????? (c3aff760, d883f8c8, 0, f9c15e50, c, c3aff690)
f9c4f278 ???????? (d6d15bc0, d883f8c8, d884f5a0, f0ac6f08, 4, c3aff720)
fa0a23a4 ???????? (d884f5a0, d883f8c8, d6d15bc0, f9c15e50, c, c3aff7e8)
f9c30f48 ???????? (d884f5a0, b6, 82bb, f9c16250, 82b9, 0)
f9c05750 ???????? (c3aff8dc, b8, f373459c, f9c15e50, c, c3aff7e8)
f9c05750 ???????? (c3aff98c, b6, 0, f9c16250, c, c3aff868)
f9c05804 ???????? (c3aff9fc, d68129b0, 0, f9c15e50, c, c3aff920)
f9c4947c ???????? (d68129b0, d8808160, f81fc000, f0818618, 8, c3aff9b8)
fa114558 ???????? (d8808160, db3d24f0, f3745c98, f9c15e50, 8, c3affab0)
fa14b064 ???????? (d8808160, f096a2a0, f81fc000, f0818618, d8808160, 9)
f9cd2c38 ???????? (c3affb9c, c3affcf0, f3745c98, f9c15e50, 8, c3affab0)
f9c0010c ???????? (c3affc28, c3affe90, a, f112bc70, 4, c3affb40)
fed5bcf8 __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (c3affe88, c3affcf0, c3affda8, a886f0, a886f0, c3affd00) + 27c
fee4a3e4 __1cJJavaCallsMcall_virtual6FpnJJavaValue_nLKlassHandle_nMsymbolHandle_4pnRJavaCallArguments_pnGThread__v_ (ff182000, a88c98, c3affd9c, c3affd98, c3affda8, a886f0) + 164
fee5d5a8 __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_ (c3affe88, c3affe84, c3affe7c, c3affe74, c3affe6c, a886f0) + 6c
fee6e8f8 __1cMthread_entry6FpnKJavaThread_pnGThread__v_ (a886f0, a886f0, c69ff8, a88c98, 31a08c, fee67ed8) + 128
fee67f00 __1cKJavaThreadDrun6M_v_ (a886f0, 78, 40, 0, 40, 0) + 284
fee643e0 _start (a886f0, c3b00000, 0, 0, fee642ac, 1) + 134
ff3404f4 _lwp_start (0, 0, 0, 0, 0, 0)

----------------- lwp# 123 / thread# 123 --------------------
ff341a08 _write (1, c3afab10, 6d, 0, 0, 22) + c
fed54288 JVM_Write (1, c3afab10, 6d, 6d, c3afab10, daa519f8) + 68
fe85e348 ???????? (c3afab10, c3afcbd0, c3afcbcc, 0, 6d, 22)
fe85e1c8 Java_java_io_FileOutputStream_writeBytes (a88784, c3afcbd0, c3afcbcc, 0, 6d, daa519f8) + 34
fa1e1608 ???????? (d8b55ad0, d8a5f6e8, 0, 6d, 0, c3afce50)
fa1e3dc0 ???????? (d8b55a70, daa726a8, 0, 6d, d351afb0, daa519f8)
f9dfae70 ???????? (d88512c8, daa726a8, 0, 6d, 0, c3afce50)
fa033244 ???????? (daa63ed8, d351afb0, da845a80, 0, d351afb0, daa519f8)
fa5fab54 ???????? (daa63ed8, d353dca0, da735c30, d351af98, 0, c3afce50)
fa599d40 ???????? (da845a80, d351afb0, da845a80, 0, d351afb0, daa519f8)
fa5977b8 ???????? (da845a80, d351afb0, da735c30, d351af98, 0, c3afce50)
fa4945b8 ???????? (2, 0, da845a80, 0, d351afb0, daa519f8)
fa5b98ec ???????? (db112a08, f1c8e7a0, da735c30, d351af98, 0, c3afce50)
fa1a906c ???????? (db146dd8, f42a9fd0, d351af98, 1, f4262f28, f81fc000)
f9c45dc8 ???????? (db146dd8, b7, 0, f9c152a0, 8, c3afcee0)
f9c05804 ???????? (c3afd05c, b6, 0, f9c16118, c, c3afcf60)
f9c05804 ???????? (c3afd0e4, b6, 0, f9c15e98, 10, c3afcff0)
f9c05804 ???????? (c3afd174, b6, 0, f9c15e98, 10, c3afd078)
f9c05804 ???????? (c3afd1dc, d34e93e0, 0, f9c15e98, 10, c3afd108)
f9c63804 ???????? (d34756e0, d34e83d8, 0, f9c, 10, c3afd190)
fa4bd36c ???????? (f9c, d34e93e0, f96, f95, d, c3afd268)
f9c63e4c ???????? (d34e83a8, b6, c3afd338, 1, 4, 0)
f9c05804 ???????? (c3afd338, f2ee27c0, 0, f9c15e98, 10, c3afd268)
f9c46bb8 ???????? (d34e9388, f0866a58, f2ee27c0, 1, 4, c3afd2f8)
fa6c56d8 ???????? (d34e9388, f2ee27c0, 2f, f0866a58, 0, c3afd478)
fa6c6c70 ???????? (d34e9388, f0866a58, f2ee27c0, 1, 10, c3afd540)
fa6ccbb8 ???????? (d34e9bc0, 0, 1, f0866a58, 8, c3afd478)
fa67bf28 ???????? (d34e9bc0, d3502ac0, d34e82d8, 1, 10, c3afd540)
fa67c168 ???????? (d34e82d8, e3c6b060, f0846b40, f0866a58, 8, c3afd478)
fa64773c ???????? (d3502ab0, d3502ac0, d34e82d8, 1, 10, c3afd540)
f9dc820c ???????? (d3502ab0, b6, c3afd628, f9c15e98, 10, 0)
f9c05804 ???????? (c3afd634, b6, 0, f9c15e98, 10, c3afd540)
f9c05804 ???????? (c3afd6b4, b6, 0, f9c15e98, 10, c3afd5c8)
f9c05804 ???????? (c3afd73c, b8, 0, f9c15e50, c, c3afd650)
f9c05804 ???????? (c3afd7c4, b6, 0, f9c16298, c, c3afd6d8)
f9c05804 ???????? (c3afd844, b6, 0, f9c15e98, c, c3afd760)
f9c05804 ???????? (c3afd8cc, b7, 0, f9c15e50, 8, c3afd7e0)
f9c05804 ???????? (c3afd95c, b6, 0, f9c160d0, c, c3afd868)
f9c05804 ???????? (c3afd9e4, b6, 0, f9c15e98, c, c3afd8f8)
f9c05804 ???????? (c3afda74, f4405ee8, 0, f9c15e50, 8, c3afd978)
f9c05a8c ???????? (c3afdb14, f45c37a0, 0, f9c163d0, c, c3afda08)
f9c05a8c ???????? (c3afdba4, f47f5e30, 0, f9c16420, 14, c3afdaa8)
f9c05a8c ???????? (c3afdc34, b6, 0, f9c163d0, 8, c3afdb38)
f9c056e4 ???????? (c3afdcd4, b6, 0, f9c15e50, c, c3afdbd0)
f9c05804 ???????? (c3afdd64, f4783158, 0, f9c15e50, c, c3afdc70)
f9c059fc ???????? (c3afddfc, b7, 0, f9c163d0, c, c3afdcf0)
f9c05804 ???????? (c3afdea4, b6, 0, f9c160d0, 8, c3afdd90)
f9c05804 ???????? (c3afdf3c, b6, 0, f9c15e50, 8, c3afde30)
f9c05774 ???????? (c3afdfc4, b6, 0, f9c15e50, 10, c3afded0)
f9c05774 ???????? (c3afe064, b6, 0, f9c15e50, c, c3afdf58)
f9c05804 ???????? (c3afe10c, b7, 0, f9c15e50, 14, c3afdfe8)
f9c05804 ???????? (c3afe19c, b6, 0, f9c160d0, 10, c3afe0a0)
f9c05804 ???????? (c3afe22c, b7, 0, f9c16250, 10, c3afe130)
f9c05804 ???????? (c3afe2cc, f4591318, 0, f9c160d0, 10, c3afe1c0)
f9c059d8 ???????? (c3afe374, b8, 0, f9c163d0, 8, c3afe260)
f9c05804 ???????? (c3afe40c, b6, 0, f9c16250, 4, c3afe308)
f9c05750 ???????? (c3afe4a4, b6, 0, f9c15e50, 8, c3afe398)
f9c05804 ???????? (c3afe53c, b6, 0, f9c15e98, 8, c3afe440)
f9c05804 ???????? (c3afe5e4, b6, 0, f9c15e98, 8, c3afe4c0)
f9c05750 ???????? (c3afe684, f43eee20, 0, f9c15e50, 8, c3afe580)
f9c059d8 ???????? (c3afe6e8, daa8e080, 0, f9c163d0, 8, c3afe618)
f9c30d40 ???????? (ddb9b518, daa8e080, d1afafe8, f9c15e50, c, c3afe6a0)
fa026230 ???????? (db146b88, daa8e080, d1afafe8, f9c16250, c, 0)
fa0182bc ???????? (daf02128, daa8e080, d1afafe8, f9c15e50, c, c3afe7b0)
f9c30f48 ???????? (daf02128, b6, c3afe89c, f9c16250, c, 0)
f9c05750 ???????? (c3afe8a4, b8, 0, f9c15e50, c, c3afe7b0)
f9c05750 ???????? (c3afe92c, b6, 0, f9c16250, c, c3afe840)
f9c05750 ???????? (c3afe990, d1afaff8, 0, f9c15e50, 4, c3afe8c0)
f9c4f278 ???????? (d1afaff8, daa8e5c0, d1b0b920, d1afb0f0, 4, c3afe950)
fa7400a0 ???????? (61a80, d1afaff8, 1, daa00d48, 2710, 34)
f9c30b04 ???????? (daf9f1a8, f441b8a8, c3afeb54, f9c163d0, f441b8a8, f441bd18)
f9c059d8 ???????? (c3afeb54, f45f12b0, 0, f9c163d0, 8, c3afea48)
f9c059d8 ???????? (c3afec1c, f0d5e7d8, 0, f9c163d0, 4, c3afead8)
f9c059d8 ???????? (c3afecac, f43eee20, 0, f9c16420, 10, c3afeb98)
f9c059d8 ???????? (c3afed54, f43eef20, 0, f9c163d0, 8, c3afec40)
f9c059d8 ???????? (c3afedf4, b6, 0, f9c163d0, 8, c3afece0)
f9c05750 ???????? (c3afee7c, b6, 0, f9c15e50, 8, c3afed88)
f9c05750 ???????? (c3afef04, b6, 0, f9c15e50, c, c3afee18)
f9c05750 ???????? (c3afef8c, b6, 0, f9c15e50, 14, c3afee98)
f9c05750 ???????? (c3aff01c, b6, 0, f9c15e50, 14, c3afef20)
f9c05750 ???????? (c3aff0ac, f43e7f08, 0, f9c15e50, c, c3afefb0)
f9c059d8 ???????? (c3aff13c, f429d368, 0, f9c163d0, c, c3aff048)
f9c059d8 ???????? (c3aff1e4, b6, 0, f9c15e50, 10, c3aff0c8)
f9c05804 ???????? (c3aff29c, b6, 0, f9c15e50, c, c3aff178)
f9c05804 ???????? (c3aff33c, b6, 0, f9c15e50, c, c3aff210)
f9c05804 ???????? (c3aff3cc, b6, 0, f9c15e50, c, c3aff2d8)
f9c05804 ???????? (c3aff464, b6, 0, f9c15e50, c, c3aff358)
f9c05804 ???????? (c3aff4f4, f1a8a5c8, 0, f9c15e50, c, c3aff400)
f9c05a8c ???????? (c3aff58c, f084f340, 0, f9c163d0, c, c3aff490)
f9c059d8 ???????? (c3aff674, b6, 0, f9c163d0, 4, c3aff518)
f9c05804 ???????? (c3aff6fc, b6, 0, f9c15e50, 10, c3aff608)
f9c05804 ???????? (c3aff760, d883f8c8, 0, f9c15e50, c, c3aff690)
f9c4f278 ???????? (d19577a8, d883f8c8, d884f5a0, f0ac6f08, 4, c3aff720)
fa0a23a4 ???????? (d884f5a0, d883f8c8, d19577a8, f9c15e50, c, c3aff7e8)
f9c30f48 ???????? (d884f5a0, b6, 786b, f9c16250, 7869, 0)
f9c05750 ???????? (c3aff8dc, b8, f375bd34, f9c15e50, c, c3aff7e8)
f9c05750 ???????? (c3aff98c, b6, 0, f9c16250, c, c3aff868)
f9c05804 ???????? (c3aff9fc, d1a10f80, 0, f9c15e50, c, c3aff920)
f9c4947c ???????? (d1a10f80, d8808160, f81fc000, f0818618, 8, c3aff9b8)
fa114558 ???????? (d8808160, d1a10f80, f3745c98, f9c15e50, 8, c3affab0)
fa14b064 ???????? (d8808160, f096a2a0, f81fc000, f0818618, d8808160, 9)
f9cd2c38 ???????? (c3affb9c, c3affcf0, f3745c98, f9c15e50, 8, c3affab0)
f9c0010c ???????? (c3affc28, c3affe90, a, f112bc70, 4, c3affb40)
fed5bcf8 __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (c3affe88, c3affcf0, c3affda8, a886f0, a886f0, c3affd00) + 27c
fee4a3e4 __1cJJavaCallsMcall_virtual6FpnJJavaValue_nLKlassHandle_nMsymbolHandle_4pnRJavaCallArguments_pnGThread__v_ (ff182000, a88c98, c3affd9c, c3affd98, c3affda8, a886f0) + 164
fee5d5a8 __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_ (c3affe88, c3affe84, c3affe7c, c3affe74, c3affe6c, a886f0) + 6c
fee6e8f8 __1cMthread_entry6FpnKJavaThread_pnGThread__v_ (a886f0, a886f0, c69ff8, a88c98, 31a08c, fee67ed8) + 128
fee67f00 __1cKJavaThreadDrun6M_v_ (a886f0, 78, 40, 0, 40, 0) + 284
fee643e0 _start (a886f0, c3b00000, 0, 0, fee642ac, 1) + 134
ff3404f4 _lwp_start (0, 0, 0, 0, 0, 0)

Another thing to check is for any other processes that are running at the same time on the server.

We faced a situation in the past, when backup processes ran at a fixed time on a particular directory, and caused CPU spikes. This was due to there being a huge number of old files in that directory.

We ran a du on the /data filesystem and as soon as it hits /data/software/xyz/log it stops dead and the CPU goes up to 0% free. This filesystem is used for deployments as the name implies. The reason was when the backup hits this directory, it will attempt to do an ls or something similar, right? As it encounters the 188994 logs that were there, it completely clogs up the CPU trying to process them. This is what is causing the CPU spikes. Whenever anyone or anything accesses this filesystem, everything stops.

On the Satyam fiasco

2009-01-08T23:16:00.000-08:00

What a change a day makes.

On Monday 5th Jan, the buzz was of a possible merger between Tech Mahindra and Satyam - and the lunchtime discussions were on swap ratios, cashless unions, share value pre and post merger.

On Tuesday the news of Mr Ramalinga Raju's fraud broke - and suddenly everyone is left gobsmacked at the enormity of the scam.

What exactly did Merrill Lynch find out about the missing cash? Is this what forced Raju's hand and made him confess?

Is the money really non-existent - or as the auditors Price Waterhouse claim - all was in order in Sept 2008 - meaning money was siphoned out afterwards?

If Satyam and PW officials are not commenting to SEBI, will the Merill Lynch folks have a better opinion for investigations to begin?

Coupled with the oil company executives strike which has forced petrol pumps running dry, suddenly #Mumbai is off the front pages.
However, the true tragedy is for the employees now with salaries being held back for 2 months and finding it difficult to get hired. Raju and his group can go underground and leave it to the "law of the land"to drag on a few 30 years or so.

Chasing SRK

2008-12-18T04:00:00.000-08:00

Last month my cousins were over from the UK. So as part of the Mumbai tour, we hung outside Mannat - Shah Rukh Khan's villa at Bandstand- for a dekko. To no avail - all we got was overhearing one security guy telling another that "Saab abhi nikala hai. Dhyaan rakhna".

We did however get a peak at his fleet of BMWs from the gate of the complex (the one on the side road which leads up to Mount Mary Church). This led rise to us discussing the newspaper report that SRK's cars are all BMWs numbered 555.

Moving on from Bandstand, we went off for dinner and a drive later along Carter Road around 11:30 pm.

Suddenly a vehicle zoomed past us - and my cousin quite casually mentioned - "Hey there's a silver BMW ending in 555 - Didnt you say SRK drives in one of those?"

"Oh is it really him?"

"Let's follow that car"

So we revved up behind the car - honking off a rickshaw and everything else that appeared in between. Getting closer - we saw the car led by a Gypsy with blinking lights and 3 security guys. Also the BMW was a 7-series sedan - so good chances that this was SRK's car with him in it.

We caught up soon enough (or rather they got bogged down in the traffic on Perry Road) and we finally stopped at the red signal - parked parallel to this car. My wife and cousins waved and danced like true mad fans - yelling "Shah Rukh Shah Rukh" - hoping he'll pull down the dark windows and wave back.

Well - the story ends on that disappointing note. After probably observing us for some time (and ensuring we're just crazy fans and not a threat - sure the security recorded my car number as well) the BMW and Gypsy cars cut through the red signal and drove off. We thought of following but noticed the 2 traffic policemen standing just ahead of the signal.

We drove back home all hugely disappointed - "Arrey Hum Bhi Santrowale hai *" so he could have allowed us a smile or a wave.

* - SRK is the brand ambassador for Hyundai Santro and the tagline for the earlier ads was Arrey Santrowale hai

And were we behind the wrong car?
Luckily a quick Google search on the car number MH 02 AU 555 solved that one. I got the photo below on a blog site.

The Sachin Effect - Ind v England Chennai 2008

2008-12-16T13:52:00.000-08:00

The day after Sachin Tendulkar did it. Helped India chase down 387 and hit a boundary on 99 to make a century as well as the winning runs.

As the last 4 was hit, Anil - my Code Red* partner-in-crime instantly messaged me - "Abhi 2 action point jyaada mila to bhi chalega"

Yes - that's the effect Sachin has on us. The feel-good factor when he plays well and India wins.
What's happened for Indian cricket on Dec 15th 2008 is the sort of thing you feel you'll never get to watch ever. SRT batting a fourth innings century and India winning the game.
Come on - break Lara's 400* too before you retire !

Congrats also to Yuvraj who stopped hitting the boundaries at 80* and leaving enough runs for SRT to get to his hundred.
Thanks to Monty Panesar and Swann for not bowling 4 wides to finish the match - and actually leaving 4 runs for India to bat at the end.
Congrats to the people(match referee?) who selected Sehwag for the deserved Man-of-the-Match and didnt get senti enough to give it to Sachin
The emotional scenes when all the ground staff swamped SRT after the game ended. Truly touching.
Poignant that Sachin dedicated his innings to Mumbai - as his RBS ad affirms "I play for India now more than ever"

And finally this classic comment on the BBC forum during the chase
"my beloved England are heading for another disappointing flat ending. We couldn't kill off a half dead runover cat with a placard in it's hand saying "do us a favour mate and kill me off"..

(*Note - My current project is in a spot of bother, hence Code Red)

Simultaneous Solaris T2000 core dumps on local zones

2008-11-25T02:22:00.000-08:00

Issue:

Our applications run on Sun T2000 servers which are configured with 4 or 5 Local zones on each Global Zone.

17th November 05:00 there was an issue on one Global Zone which caused 5 separate and simultaneous core dumps to get created on the server. One core in each Local zone.

Impact:
The 5 processes which crashed on each Local zone were

LZ 15 – core created by the wlshell java process. Low impact
LZ 27 – one core created by wlshell java processes. Low impact.
LZ 25 – core created by HM server java process. Was manually restarted today. Medium impact
LZ 26 – core created by PA server java process. This was automatically restarted by the HM. High impact.
LZ 37 – core created by HM server java process. Was manually restarted today. Medium impact

All 5 crashes above were in Library=/platform/sun4v/lib/libc_psr.so.1

Each core exits with a hs_err_.log file - which contains the foll details

An unexpected exception has been detected in native code outside the VM.
Unexpected Signal : 10 occurred at PC=0xFF2709F0
Function=_memset+0x70
Library=/platform/sun4v/lib/libc_psr.so.1

Current Java thread:
at java.lang.Thread.start(Native Method)
- locked <0xf1f3a378> (a java.util.logging.LogManager$Cleaner)
at java.lang.Shutdown.runHooks(Shutdown.java:126)
at java.lang.Shutdown.sequence(Shutdown.java:165)
at java.lang.Shutdown.exit(Shutdown.java:210)
- locked <0xf5998498> (a java.lang.Class)
at java.lang.Runtime.exit(Runtime.java:90)
at java.lang.System.exit(System.java:715)
at ConsoleScriptRunner.main(ConsoleScriptRunner.java:64)

Investigations

1. Seems to be running out of swap at the time of the crash. Memory: 31.9G real, 4.0G free, 37.7G swap in use, 2.4G swap free
Update: added 50 G swap on 19/11 - Memory: 31.9G real, 3.4G free, 38.0G swap in use, 52.8G swap free

2. Application Support is identifying what the wlshell process does – it seems to be a Weblogic server monitoring script running every 10 minutes under cron.

3. Library=/platform/sun4v/lib/libc_psr.so.1 Are there any known issues with this library on solaris 10 or T2000? Support to query SUN

The libc_psr libraries implement platform-specific, optimized versions of block copy and move routines from libc, such as memcpy(). On UltraSPARC machines, these routines are coded in assembler, and use block load and store ASI's, prefetch, and other tricks for better performance.

Root Cause

Sun identified that the issue was related to known bugs in JDK 142_03 and running an old version of libc_psr.so

The ( a) type error as seen in hs_err_pid6216_18111030.log the error was a know java bug with the j2sdk1.4.2_03 and the bug is identified as :

Bug ID: 4927116 fixed in 1.4.2_04 Synopsis Regression: 1.4.2 JVM core dumps in ClassLoader.defineClass0()

Actions

1. Immediate action is to upgrade to JDK 142_18

2. Also identified that all 5 core dumps were created by sh scripts running under crontab. Further analysis showed that there were cron jobs running every 15 mins on each of the zones - and this puts a load on system resources especially around memory consumption in the JVM and sufficient unhanded exceptions between the java virtual machine and the operating system could cause the then dead locked JVM to terminate .

Thus the 2nd resolution was to offset the start time of each cron job - to try and ensure no two jobs run on the same Global Zone at the same time.

Old Timings:

0,15,30,45 * * * * /export/home/Handlers.sh > /tmp/Handcheck
0,15,30,45 * * * * /export/home/Average_response_time.sh > /tmp/Avgcheck
0,15,30,45 * * * * /export/home/Msg_In_and_Out_count.sh > /tmp/Mescheck
0 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23 * * * /log/zipper.sh

New Timings:

1,16,31,46 * * * * /export/home/Handlers.sh > /tmp/Handcheck
2,17,32,47 * * * * /export/home/Average_response_time.sh > /tmp/Avgcheck
3,18,33,48 * * * * /export/home/Msg_In_and_Out_count.sh > /tmp/Mescheck
4 0,2,4,6,8,10,12,14,16,18,20,22 * * * /log/zipper.sh

Also knocking off unnecessary cron jobs from the box to reduce the load.

Car journey to Shirdi

2008-10-22T02:30:00.000-07:00

We travelled to Shirdi last weekend - took my own car along for a good self-driven road trip.

The road is quite straightforward - from Mumbai take the Eastern Express Highway to Thane and Bhiwandi. This is the NH3 to Nashik, but we took a route avoiding Nashik. Left from home at 7:30 am.
First stop was for snacks at Manas Resorts Igatpuri at around 10:30 am. The resort has been nicely done-up and looks a good spot for a weekend getaway a la Fariyas.

7 km after Igatpuri is the town of Ghoti which has a bypass road to Sinnar, which avoids Nashik completely. There is a board on the highway pointing to a right turn to Ghoti. The off-road leads through a market for about 2 kms, but ask for directions to Sinnar/Shirdi if confused. It leads to a railway crossing and a toll booth later - and then the bypass road smoothens out.
This route was a scenic drive through some unspoiled lands. The photos show the windmills and lakes seen along the Ghoti-Sinnar bypass road. Another advantage was no high-speed overtakers on this route, since the road is well-hidden. There are no boards informing you of this route, and you get 60 kms of peaceful leisurely driving.

At Sinnar, you rejoin the SH10 which is a straight road to Shirdi. Back to the superfast buses and crazy drivers. We reached Shirdi at 2:15 pm and quickly found a hotel for ourselves.
The Hotel Sai Baba International - which is right opposite the main gate of the temple complex. The rooms were adequate, 3 star hotel with clean facilities.

Road distances:
Andheri East - 0 Km
Ghatkopar on EEH - 13 Km
Shahapur - 74 Km
Igatpuri - 124 Km
Ghoti - 131 Km
Sinnar - 190 Km
Shirdi - 250 Km

The journey took 7 hours on the way out, but we spent 1.5 hours in breaks.
The return was quicker since we knew the route well, 6 hours in all including lunch at Foodplaza beyond Igatpuri (Avoidable!) .

St Alphonsa canonized

2008-10-14T23:54:00.000-07:00

There's been lots of news recently over Sr. Alphonsa - a Catholic nun from Bharananganam, in Kottayam district, Kerala being proclaimed a Saint on Oct 12th 2008. She will now be known as St Alphonsa of the Immaculate Conception. This is newsworthy since she is the second saint from India and the first woman.
Many would have expected Mother Teresa to be the first woman from India - but things dont move so fast in the Vatican - St Alphonsa has been canonized 62 years after her death.

I was watching the news on Sunday - CNNIBN I think it was - an over-excited reporter outside Mahim church had found a Kerala-ite lady who "had actually been to the church of St Alphonsa" and was asked in true AajTak style - "How do you feel about it?" "Kaise mehsoos ho raha hai"types.

As over-the-top as that was - it truly hit me then how proud a moment it was for people like myself and my family - those who have a small (and I mean really small) association with this saint, the town and the church. Bharananganam is a town 4 kms away from my mom's native place of Edamattom in Kottayam district, Kerala. My mom has in fact studied in the same convent school attached to this church - and when she was a student, there was the buzz around the church and convent - that of Sr Alphonsa - the recently passed away nun who led a simple and pious life. She was a model for all other girls to follow and this was part of the school's christian studies too.

Sr Alphonsa was beatified by Pope John Paul II on his India visit in 1986. [Beatification is the penultimate step in the process of sainthood]. This led to the mass following there is now - the hope that she will be declared a saint one day - and that she will intercede and miracles can happen. Throughout my childhood, we used to regularly visit the church on each visit to Kerala. This has continued till today - and is one of the things we do without fail.

I think the difference now that she has become a Saint - means Christians from all over India and even abroad will now make this a pilgrimage place. Similar to St Francis Xaviers church in Goa and Vailankanni in Tamil Nadu. So the roads have been widened, the town painted. On my last visit, the church had been restored and the gardens had been replanted completely. The church becomes a pilgrimage centre and the town prospers too. Since this is in Kottayam, a visit to the church might soon be added on tour itineraries along with the Kumarakom lake and houseboat rides.
I do hope things remain as they were - small and simple. Not turning into the commercial circus most religious places become.

Well time will tell.