Installing multiple external jars into a local maven repository.

Suppose you have a variety of dependency jars you need to install into your local repository - Note: These arent artefacts produced by your build but external jars. Well, no shortcuts here. From a recent reply I gave on Stackoverflow - You'll have to do it via a script, the long and boring way. This is what we use - modifying the values in each of the `set` statements

    set GROUP_ID=someNamespace
    set ARTIFACT_ID=myartifact
    set VERSION=1
    set COMPONENT=%ARTIFACT_ID%-%VERSION%
    set FILEPATH=D:\my.jar
    
    call mvn install:install-file ^
     -DgroupId=%GROUP_ID% ^
     -DartifactId=%ARTIFACT_ID% ^
     -Dversion=%VERSION% ^
     -Dfile=%FILEPATH% ^
     -Dpackaging=jar ^
     -DgeneratePom=true
    
    set GROUP_ID=someNamespace2
    set ARTIFACT_ID=myartifact2
    set VERSION=1
    set COMPONENT=%ARTIFACT_ID%-%VERSION%
    set FILEPATH=D:\my2.jar
    
    call mvn install:install-file ^
     -DgroupId=%GROUP_ID% ^
     -DartifactId=%ARTIFACT_ID% ^
     -Dversion=%VERSION% ^
     -Dfile=%FILEPATH% ^
     -Dpackaging=jar ^
     -DgeneratePom=true

Rooting HTC Wildfire and installing Cyanogenmod

There are many guides out there, this is a diary of events and some errors you might run into and resolutions which worked for me.

Phone Model HTC Wildfire A3333 running on Android 2.2.1 (Froyo)

Began with HBOOT 1.01.0002

Note: To find your HBOOT version -

Switch the phone off, then hold the volume down button and while holding press the power button. At the top of the screen it will show the HBOOT version. To exit this screen, press the power button to select fastboot and then use the volume down to select reboot.
The One-click guide given at http://www.brighthub.com/mobile/htc/articles/100819.aspx did not work initially.

It ran successfully up to step 11 out of 11, when running Revolutionary it gets to a screen on the tool with the message stuck at "Waiting for device"

So I had to do this the older way downgrading to HBOOT 1.01.0001 followed by switching the phone to S-OFF mode

The complete guide which works for me is this one

http://www.aritrasen.com/2011/09/17/how-to-root-htc-wildfire-2-2-1-and-install-cyanogenmod/


Further errors that appear along the way and their solutions:

1. Revolutionary stuck at "Waiting for device"

The solution which worked after consulting various forums, was to install latest version of HTC Sync on the PC and then uninstall it, leaving the HTC Drivers on the PC.

Under Control Panel > Add/Remove Programs you should see the HTC drivers shown in the image below.



2. After this it proceeds to the point where Revolutionary says "Waiting for root"

This takes forever and does nothing.

Of course this is because I thought one of the steps was optional - that of downgrading from Froyo(2.2.1) to Eclair(2.1). Turns out, I do need to do that - once that is done, Revolutionary goes ahead and gives the Success message.


3. After that you can install Cyanogenmod. Ensure you install 7.1.0.1 and not 7.1.0.

7.1.0 is buggy and as soon as the phone reboots, it gives a "Force Close" message with no status / notification bar. It also does not show any incoming phone calls - you can hear the ringtone but there is not way to accept the call.

All of this is fixed in 7.1.0.1 which is downloadable as update-cm-7.1.0.1-buzz-signed.zip

After that, surprise surprise, there's no GMail, Market or Maps. These are not part of Cyanogenmod. These are available as a separate zip from http://wiki.cyanogenmod.com/wiki/Latest_Version/Google_Apps.

I used gapps-gb-20110828-signed.zip for Wildfire but still had to install Google Maps separately from the Market.

After this, I uninstalled some of the apps I didnt want to use in the Cyanogenmod ROM.

A list of what is safe to remove and what is not is given at the

Cyanogenmod wiki at http://wiki.cyanogenmod.com/wiki/Barebones

Follow the instructions and play around with it.

I later upgraded to CM 7.2 using this ROM from here

For those used to HTC's Predictive Dialer, the native CM one kind of sucks. Use the Touchpal Contacts smart dialer app from the Market. http://forum.xda-developers.com/showthread.php?t=817128

With 7.2 I've seen my battery charges quicker than before (with HTC ROM) and the interface is much quicker too.

Dont' forget to add some Lockscreen Gestures

More cool things to do are here

Additional reference links:

http://androidforums.com/wildfire-tips-tricks/348619-easy-ways-improve-htc-wildfire-speed.html

http://androidforums.com/wildfire-all-things-root/459656-what-do-when-rooted.html


http://unrevoked.com/rootwiki/doku.php/public/adb_in_recovery

Weblogic - Accessing Server Runtime details using JMX

We had a requirement to access certain runtime info on the Weblogic server to carry out some admin tasks.

One way to do this is via the JMX API, which in Weblogic is available as an extensive set of MBeans

There are plenty of tutorials and examples of using these on the net.

Now within your Weblogic domain, if you run as part of a Cluster then you would access Domain information on the Admin server if you need to identify the running servers in the cluster.

This uses

weblogic.management.mbeanservers.domainruntime.DomainRuntimeServiceMBean

object which is bound only on the Admin server.
One full example of this is at http://middlewaremagic.com/weblogic/?p=210

If you try this on a managed server directly, you will get this error

java.io.IOException: Unable to resolve 'weblogic.management.mbeanservers.domainruntime'. Resolved 'weblogic.management.mbeanservers'
        at weblogic.management.remote.common.ClientProviderBase.makeConnection(ClientProviderBase.java:195)
        at weblogic.management.remote.common.ClientProviderBase.newJMXConnector(ClientProviderBase.java:83)
        at javax.management.remote.JMXConnectorFactory.newJMXConnector(JMXConnectorFactory.java:338)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:247)

So if you do not have the Admin server IP address and port, how can you still identify the other members in the cluster?

A use case for this is if you want to run and access Cluster details from one of the managed servers. But you only have the current running server IP and Port and not that of the Admin server - which is usually the case

There is an MBean for this which is available on the individual managed server as well,
weblogic.management.mbeanservers.runtime.RuntimeServiceMBean

Using this you can get the details of the Admin server and then from there get access to the DomainRuntimeServiceMBean.

A working code example is below

This runs from a JSP where we are able to access the server IP and port via request variables

<%@page import="javax.xml.bind.DatatypeConverter"%>
<%@page import="java.io.IOException"%>
<%@page import="java.net.MalformedURLException"%>
<%@page import="java.util.Hashtable"%>
<%@page import="javax.management.MBeanServerConnection"%>
<%@page import="javax.management.MalformedObjectNameException"%>
<%@page import="javax.management.ObjectName"%>
<%@page import="javax.management.remote.JMXConnector"%>
<%@page import="javax.management.remote.JMXConnectorFactory"%>
<%@page import="javax.management.remote.JMXServiceURL"%>
<%@page import="javax.naming.Context"%>
<%@page import="java.net.URLConnection"%>
<%@page import="java.net.HttpURLConnection"%>
<%@page import="java.net.URL"%>
<%@page import="sun.misc.BASE64Encoder"%>
<%@page import="java.io.InputStream"%>
<%@page import="javax.management.MBeanServer"%>
<%@page import="javax.management.ObjectName"%>
<%@page import="javax.naming.InitialContext"%>

 MBeanServer mBeanServer = null;
 InitialContext ctx = null;
 String administrationURL = null;
 int adminPort = 0;
 try {
  ctx = new InitialContext();
  mBeanServer = (MBeanServer) ctx.lookup("java:comp/env/jmx/runtime");
  
  //Get Admin Server and Port
  ObjectName runtimeService = new ObjectName(
     "com.bea:Name=RuntimeService,Type=weblogic.management.mbeanservers.runtime.RuntimeServiceMBean");
  String managedServerName = (String) mBeanServer.getAttribute(runtimeService, "ServerName");
  ObjectName msServerRuntime = new ObjectName("com.bea:Name="+ managedServerName + ",Type=ServerRuntime");
  administrationURL = (String) mBeanServer.getAttribute(msServerRuntime, "AdminServerHost");
  adminPort = (Integer) mBeanServer.getAttribute(msServerRuntime, "AdminServerListenPort");
  System.out.println(administrationURL + adminPort);
 } catch (Exception ex) {
  System.out.println("Caught Exception while fetching Admin Server information : "+ ex);
  ex.printStackTrace();
 } finally {
  if (ctx != null) {
   try {
    ctx.close();
   } catch (Exception ex) {
    ex.printStackTrace();
   }
  }
 }

 //Connect via JMX using Admin Server credentials
 String protocol = "t3";
 Integer portInteger = Integer.valueOf(request.getServerPort());
 int port = portInteger.intValue();

 String jndiroot = "/jndi/";
 String mserver = "weblogic.management.mbeanservers.domainruntime";
 JMXServiceURL serviceURL = new JMXServiceURL(protocol, administrationURL, adminPort, jndiroot + mserver);
 Hashtable h = new Hashtable();
 h.put(Context.SECURITY_PRINCIPAL, weblogicuser);
 h.put(Context.SECURITY_CREDENTIALS, weblogicpassword);
 h.put(JMXConnectorFactory.PROTOCOL_PROVIDER_PACKAGES,"weblogic.management.remote");
 JMXConnector connector = JMXConnectorFactory.connect(serviceURL, h);
 MBeanServerConnection connection = connector.getMBeanServerConnection();

 ObjectName domainRuntimeService = new ObjectName(
 "com.bea:Name=DomainRuntimeService,Type=weblogic.management.mbeanservers.domainruntime.DomainRuntimeServiceMBean");
 ObjectName[] serverRT = (ObjectName[]) connection.getAttribute(domainRuntimeService, "ServerRuntimes");
 Hashtable server_states = new Hashtable();
 for (ObjectName ser : serverRT) {
  server_states.put((String) connection.getAttribute(ser,"Name"), (String) connection.getAttribute(ser,"State"));
 }

 ObjectName domain1 = (ObjectName) connection.getAttribute(
    domainRuntimeService, "DomainConfiguration");
 ObjectName[] cluster_list = (ObjectName[]) connection
    .getAttribute(domain1, "Clusters");
 for (ObjectName cl : cluster_list) {

  System.out.println("

 Cluster Name: "
    + (String) connection.getAttribute(cl, "Name"));
  try {
   System.out.println("

");
   ObjectName[] servers = (ObjectName[]) connection.getAttribute(cl, "Servers");
   for (ObjectName ser : servers) {
    String server_name = (String) connection.getAttribute(ser, "Name");
   
    try {
     String server_URL = (String) connection.getAttribute(ser, "ListenAddress");
     Integer server_port = (Integer) connection.getAttribute(ser, "ListenPort");
     System.out.println("Server Name: " + server_name + ", Server State: "
        + server_states.get(server_name)
        + ", :" + server_URL + ":"
        + server_port);

   
    }
   }
  }
  connector.close();
 }

Some info on Weblogic clustering

Recently this question regarding Weblogic clustering came up on Stackoverflow.

I put some time into researching the answer I provided, and thought of putting this up on the blog as well.

Question from user Russell

I've read http://download.oracle.com/docs/cd/E11035_01/wls100/cluster/overview.html and searched this topic on the internet but still had a hard time understanding some of weblogic's clustering concepts.

Can anybody confirm/correct my understandings below?

•a cluster contains one or more logical servers which can reside on one or many physical servers
•when deploying a j2ee app to a cluster, it is tied to one server in that cluster
•external users of the deployed app aren't aware of clustering
•the log file of that app is located on the server it's deployed
•if the server hosting the app fails, it's okay because the app is in a cluster and another server will pick up the work?
•if the server hosting the app fails, what happens to logging?
Maybe I got the whole concept wrong. Could anybody point me into the correct directions?

Answer

It's good to understand the concept of Domain first.

The Domain is the parent of a Cluster. It contains typically one Admin and one or more Managed servers.
Now the Cluster is a grouping of some or all of these managed servers within the domain.

This diagram here should help understanding the relation between Domains and Clusters

Once you configure a Domain and a Cluster yourself on a development environment, you'll get to know more about it.

Now here are the answers to your specific questions

•a cluster contains one or more
logical servers which can reside on
one or many physical servers

True. But let's clarify what you mean by 'logical' servers.
In the Cluster you typically have two or more Managed servers. These servers run in their own JVMs and can be started independently and serve requests independently. Each server will have a unique IP:port address, and it can be directly accessed from the browser. But these server instances can reside over multiple physical servers.

•when deploying a j2ee app to a
cluster, it is tied to one server in
that cluster

No it is not tied to one server. When you deploy a J2EE app to the Cluster, it will get deployed in turn to each server in that cluster. The JNDI is cluster-wide and each server maintains a local copy of the JNDI.

You can look up the object (say an EJB) via JNDI on the Cluster or on the individual server. Also see what types of Objects can be clustered.

•external users of the deployed app
aren't aware of clustering

True.

But in this case you should have an Apache web server or a load balancer or DNS server which takes the request from the browser, and internally maps it to one of the servers in the cluster. If you dont have any of these, you would have to define the cluster address as a DNS name or IP address for the client. See the section "Avoiding Listen Address Problems " on http://download.oracle.com/docs/cd/E13222_01/wls/docs103/cluster/setup.html#wp682940

•the log file of that app is located
on the server it's deployed

True, one weblogic log per server.

•if the server hosting the app fails,
it's okay because the app is in a
cluster and another server will pick
up the work?

Not by default, you have to configure it for failover and replication. This is a huge topic which needs separate reading

•if the server hosting the app fails,
what happens to logging?

Logging stops. You'll see some shutdown or heartbeat errors in the log, or outofmemory or whatever reason for failure. you'll have to restart the server - and logging continues in a new file (depending on your logging settings)

My original answer is here on stackoverflow for those interested.

Weblogic JDK 1.6 Error Could not reserve enough space for object heap

Weblogic 10 running on HP-UX throws this error on startup when we try to give 3 GB Xms and Xmx.

-Dweblogic.system.BootIdentityFile=/portalapp/bea/user_projects/domains/MSD4/servers/MS4/data/nodemanager/boot.properties -Dweblogic.nodemanager.ServiceEnabled=true -Dweblogic.security.SSL.ignoreHostnameVerification=false -Dweblogic.ReverseDNSAllowed=false -server -Xms3g -Xmx3g -Xmn1024m -XX:+AggressiveHeap -XX:+UseParallelGC -XX:ParallelGCThreads=20 -XX:PermSize=1024m -XX:MaxPermSize=1024m -XX:MaxTenuringThreshold=3 -XX:LargePageSizeInBytes=4m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:SurvivorRatio=8 -XX:-UseAdaptiveSizePolicy -Xloggc:/portal_logs/gclog/MS4_gclog.log weblogic.Server >

<Feb 17, 2010 10:56:06 AM> <Info> <NodeManager> <Working directory is "/portalapp/bea/user_projects/domains/MSD4">

<Feb 17, 2010 10:56:06 AM> <Info> <NodeManager> <Server output log file is "/portalapp/bea/user_projects/domains/MSD4/servers/MS4/logs/MS4.out">
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.
<Feb 17, 2010 10:56:06 AM> <Debug> <NodeManager> <Waiting for the process to die: 13663>
<Feb 17, 2010 10:56:06 AM> <Info> <NodeManager> <Server failed during startup so will not be restarted>

Additional information:

os.name = HP-UX
os.version = B.11.31

java.version = 1.6.0.04
java.vm.info = mixed mode
java.vm.name = Java HotSpot(TM) Server VM
java.vm.specification.name = Java Virtual Machine Specification
java.vm.specification.vendor = Sun Microsystems Inc.
java.vm.specification.version = 1.0
java.vm.vendor = "Hewlett-Packard Company"
java.vm.version = 11.3-b02-jre1.6.0.04-rc2
java.runtime.version = 1.6.0.04-jinteg_28_apr_2009_04_46-b00


Analysis:


Though we are using a 64-bit server which has 16 GB RAM, we are unable to assign more than 2 Gb to the Weblogic heap.

This value in the startup log shows that the JVM is a 32-bit one.

sun.arch.data.model = 32


In a 32-bit JVM – the process cannot address memory higher than 4 GB in all. But as per Sun site, in reality we cannot get more than 2G.

As per this URL: http://java.sun.com/docs/hotspot/HotSpotFAQ.html#gc_heap_32bit

Why can't I get a larger heap with the 32-bit JVM?

The maximum theoretical heap limit for the 32-bit JVM is 4G. Due to various additional constraints such as available swap, kernel address space usage, memory fragmentation, and VM overhead, in practice the limit can be much lower. On most modern 32-bit Windows systems the maximum heap size will range from 1.4G to 1.6G. On 32-bit Solaris kernels the address space is limited to 2G.

So we tuned the heap to 2 GB for now. i.e. set Xmx and Xms to 2G and that solved it.

More tuning tips on slow Weblogic 10

In the earlier parts of this article I described some technical tuning tips specific to a slow-performing Weblogic JEE server.

Another earlier post has looked at how to set the JSP check seconds values.

This is a follow-up article specifically for tuning the similar settings for Weblogic 10 - since the syntax has changed.


In weblogic.xml, ensure the following are set as below:

<wls:session-descriptor>
<wls:encode-session-id-in-query-params>true</wls:encode-session-id-in-query-params>
<wls:cookie-comment>mysessioncookie</wls:cookie-comment>
<wls:cookie-max-age-secs>-1</wls:cookie-max-age-secs>
<wls:cookie-name>JSESSIONID</wls:cookie-name>
<wls:cookie-path>/</wls:cookie-path>
<wls:cookies-enabled>true</wls:cookies-enabled>
<wls:timeout-secs>1200</timeout-secs>
<wls:cookie-domain>@MY_COOKIE_DOMAIN@</wls:cookie-domain>
</wls:session-descriptor>



Assuming that no JSPs are being hot deployed or dropped directly into the production server

<wls:jsp-descriptor>
<wls:keepgenerated>true</wls:keepgenerated>
<wls:precompile>true</wls:precompile>
<wls:page-check-seconds>-1</wls:page-check-seconds>
</wls:jsp-descriptor>


and this one

<wls:container-descriptor>
<wls:servlet-reload-check-secs>-1</wls:servlet-reload-check-secs>
</wls:container-descriptor>


Ensure all of these are inserted at the right place within the weblogic.xml to avoid any start up error messages

Check against the reference here http://download.oracle.com/docs/cd/E12840_01/wls/docs103/webapp/weblogic_xml.html

Postcards from New Zealand

I spent 2 months on a client assignment in New Zealand last year. The summer was a great time to be there and here are some pictures.



Auckland skyline




Looking up at the SkyCity Tower Auckland





Cathedral Cove in the North Island







View of Auckland from Waiheke Island







The Cruise on Milford Sound in the South Island. The view ahead is the Trans Tasman Sea.





The seals at Milford Sound




Birds at Queenstown




Lake Pukaki on the way to Mount Cook




View of Mount Cook from Lake Tekapo



The Church of Good Shephed on the shores of Lake Tekapo




View of Lake Tekapo from Mount John Observatory




Two men playing chess at Christchurch City Centre. Let me know if it's you :)

Weblogic JMS Performance Tuning Tips

Here are a couple of real-life tips on tuning JMS performance.

Problem:
The creates and sends JMS messages on an outgoing queue for consumption by another application.

It was observed that when the consuming application was offline for a period of time the number of messages that could be retained on the queue before the JVM heap was filled up was quite low. This was tested to be roughly 7000 messages, after which OutOfMemory exceptions begin to occur. Given that the consuming application could realistically be offline for a period, hence the use of an asynchronous queue, we needed to increase the number of messages that could realistically be stored in the queue.

The exception we get is shown below:

Start server side stack trace:
java.lang.OutOfMemoryError:

Start server side stack trace:
java.lang.OutOfMemoryError
        <<no stack trace available>>
End  server side stack trace
        at weblogic.rmi.internal.BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:109)
       at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:127)
       at weblogic.jms.dispatcher.DispatcherImpl_WLStub.dispatchSyncFuture(Unknown Source)
       at weblogic.jms.dispatcher.DispatcherWrapperState.dispatchSync(DispatcherWrapperState.java:286)
       at weblogic.jms.client.JMSSession.createProducer(JMSSession.java:1484)
       at weblogic.jms.client.JMSSession.createSender(JMSSession.java:1335)
...

The GC logs also show frequent Full GCs before the server goes out of memory.


Solution Steps:

1. Enabling JMS Paging

Paging had not been enabled for the queue. Despite this queue being persistent, this meant that every message was stored in the JVM memory heap in its entirety. Enabling message paging for this queue means that only the headers for paged messages are kept in memory, significantly reducing the amount heap utilized.

As the messages were being persisted via a JDBCStore to a database, this functioned as a paging store as well, however a FileStore must still be specified as the paging store for the JMS Server, or the JMS Server will not deploy at WLS server start time. This would be due to the need to cater for any non-persistent destinations when paging is enabled. If non-persistent messages are not paged, the size of this FileStore will be negligible. Paged messages still occupy some space on the memory heap as the message headers are still kept in memory.

On enabling paging for the specific queue, the test could cater for roughly 15000 messages before OutOfMemory exceptions occurred. The point at which paging began was set deliberately low to 100. Recovering from the page store does incur a certain performance cost, so in Production this was set to a more reasonable number based on the peak number of messages expected in the queue under normal conditions.

Despite the gain in number of messages that could be catered for, the heap utilisation graphs were very similar to those before paging was enabled. This showed no minor GCs, only full GCs at fairly frequent intervals.


2. JVM Settings and Garbage Collection Tuning

The untuned JVM heap size was 512Mb. This value could be increased, but test results after tuning the JVM settings indicate that this was probably more than adequate.

Examining the current JVM settings uncovered some settings that needed to be changed.

The most significant issue with the JVM settings was the NewSize value. This was set very high to 384Mb out of the total heap of 512 Mb.
A reasonable New Generation area would normally be 20-25% of the total heap size and setting it larger than the Tenured Generation area is guaranteed to cause unhealthy GC operations. In addition, it is good practice to use NewRatio rather than NewSize to avoid fixing an absolute size. A NewRatio of 3 (1:3, i.e. 25% of heap) or 4 is considered the most appropriate for WebLogic Server applications. NewSize was therefore dropped and a NewRatio was set to 4 (20% of total heap).

The SurvivorRatio value was set to a reasonable value of 3.
TargetSurvivorRatio, however, was unset meaning that the default of 50 applies. 80 would probably be a better setting, meaning that the switch between survivor spaces in the JVM heap would occur at 80% rather than 50%. The performance improvement from this should be noticeable in the frequency of minor GC, though not dramatic.

The PermSize values were high, with PermSize and MaxPermSize both set to 384Mb. These were reduced to 64Mb and 128Mb respectively, which should be more than adequate. Though these changes are unlikely to improve performance, having the PermSize set to high will needlessly consume memory.

The effect of setting a good NewRatio value was dramatic. Many minor GCs were the rule, with infrequent full GCs. 60,000 messages were added to the queue before the heap was approaching full. We ran an overnight, and somewhere between 60,000 and 70,000 messages the OutOfMemory exceptions occurred.

Scaling this up to the Production environment which has 1Gb Heap, shows that the server could easily cater for the expected load.

Dave Matthews Band - Big Whiskey and the Groogrux King

My favourite group, Dave Matthews Band released their 7th studio album - "Big Whiskey and the Groogrux King". I feel this is their best album since 1998's "Before These Crowded Streets" with them going back to their jamming style of music rather than the manufactured pop sounds of recent albums.
The name is a tribute to founding member and saxophonist LeRoi Moore who died in August 2008 and his nickname was Groogrux. A few of the songs on the album have references to LeRoi and show that the band really miss their mate. The cover portrays Moore's head on a float in the Mardi Gras procession in New Orleans.





The album begins with a short and lovely saxphone solo "Grux" - recorded by LeRoi before his death.

This is the prelude to the first song "Shake Me Like a Monkey" which is one of the outstanding tracks. The lyrics exhort a babe so hot "that God and the devil alone could not have made you up. Two must have worked as one" to "Love Me Baby Shake Me Like a Monkey". and this gem "I like my coffee with toast and jelly. But I'd rather be licking from your back to your belly". Very peppy and a kick-ass blast this song is.

"Funny the Way It Is" is a poem showing the same events has opposite effects on different people. "Hear the laughter while the children play war" and "Funny the way it is, If you think about it, One kid walks ten miles to school, another's dropping out"
The words remind me of Alanis Morissette's "Ironic" and the tune is simple and catchy with subtle violin at the start and a full blooded violin interlude in the middle with strong rock guitar riffs. I think this song also shows the band misses LeRoi with this line "Somebody's heart is broken and it becomes your favorite song". I dont like the video too much though.


The third song "Lying in the Hands of God" is a slow ballad with a simple arrangement. Hummable with reflective lyrics. This song grows on you with it's slow guitar rounding off - and you can picture yourself looking into a fading sunset with a drink in hand.

Then there's another decibel upping blast "Why I Am" - starts off with a full-on drums and guitar rolls and is definitely again a tibute to LeRoi "Why I am still here dancing with the GrooGrux King, We’ll be drinking Big Whiskey while we dance and sing" and you do want to tap along with the dance. This is surely an instant hit.

"Dive In" is my 2nd favourite song on the album - the one I loved as soon I heard it. It starts on a slow note but Dave sounds hypnotical in the lilting tune singing an ode to all things Summer - Summer games, Summer breeze, Summer girls, Summer dresses - "Kick off your shoes, dive in the empty ocean". "Wake up sleepy head, I think the suns a little brighter today". It surely is guys - after listening to this - my day has brightened. Wake up and enjoy it while it lasts cause it wont last forever - is the message here.


"Spaceman" is an okay song compared to the others. If it were on one of the weaker earlier albums, Busted Stuff [2002] or Stand Up [2005], this might have been rated higher but for me, it just makes up the numbers on this album. the lyrics are about a guy's love for his girl, even when things dont work out.

"Squirm" is quite unusual and surprising. It starts off slow but punches you when Dave gets a bit menacing and the crescendo builds as he sings "Drumbeats louder (x 4)". He gets ferocious on the "Open up your head" chorus. Parts of it almost feels like a song by U2, I can imagine Bono cooing in falsetto between the climactic violin and drum rolls, and it blends in what sounds quite Middle Eastern wailing for the finale. There's much more meaning to these lyrics with it's referring to (lack of) Faith and the differences in Religions and the inevitable End. But I love the song for it's music rather than it's philosophy.

"Alligator Pie (Cockadile)" is my favourite song on the album. It's arranged chaos is sublime - listen closely for the dog barking just before the violin in the beginning and Dave's own howling bringing in the vocals. This song will get your feet tapping with the banjo driving the entire melody. On the face of it, this is a song for his daughter Stella asking "Daddy when you going to put me in a song" since many earlier songs have referenced her twin sister Grace. (Yup, even I dont get why didnt her put her in a song earlier) But Dave's also pushes in a critical reference to the Bush administration's response to the Hurrican Katrina devestation of New Orleans. The song proceeds into a lovely cacophony with Dave almost yelping through the words. My only complaint is it ended too soon at 3:57 mins. The title word "cockadile" is the child's pronunciation of "crocodile" but I did not hear the word used in the song. Must be an in-family joke.

"Seven" begins with a heavy guitar sound almost like Pearl Jam and then moves into another jamming event. Most of the song is in falsetto with plenty of naughty lyrics. I wonder if the title itself alludes to The Seven Minutes- the novel by Irving Wallace. But it's fun to hear this all the same - and you'll start humming it in no time.

"Time Bomb" begins so similar to "The Dreaming Tree" from "Before These Crowded Streets". Soft alto saxophone and ballady tune "I'm a ticking time bomb Waiting to blow my top" and - Woah it explodes into a powerful metal guitar as if Metallica guested on this after 2:40 mins. Dave's shouting out angrily and covered by a strong trumpet "I want to pick up the pieces, Hammer in the final nail, I wanna believe in Jesus". Yes this is a dark song about death and killing and the purpose of it all. This song echoes very strongly in your heart if you have experienced loss - either by death or a failed relationship.
There are some lyrics which possibly refer to the death pf Dave's older sister, Anne, who lived in South Africa, was murdered by her husband, who subsequently committed suicide, on or around January 27, 1994. "No one would believe it
He was such a normal guy
Shake their heads and wonder why?"
I feel the subtext here is on the existence of God and whether we are controlled by a higher power "If Martians fell from the sky what would that do to God? ".

When you're recovering from that rant, the soft guitar chimes in like a refreshing glass of chilled water and thats the opening of "My Baby Blue". This is the best ballad on the album - a love song dedicated to the one true love who he seems to have lost "Holding tight to the legs of all your angels, Goodbye my love ". Talking about the blue eyes, I feel though this is about loss, it's not dedicated to LeRoi, and there are some forums suggesting this is about his sister Anne's death. In a way this seems like he is at peace with the loss that was experienced in "Time Bomb" and I personally feel this is a follow-on to Time Bomb. This is an 'easy listening' track for Pop fans.

"You and Me" is another simple and easy ballad. This is probably a dedication to his wife - "When the kids are old enough we're gonna teach them how to fly. You and me together, We can do anything baby". This is mostly in acoustic guitar and rounds off the album on a soft note.


Yes - This is a lovely collection and took some time coming.

Cheers to Dave, LeRoi, Stefan Lessard, Carter Beauford, Boyd Tinsley, Jeff Coffin and Tim Reynolds with a Big Whiskey in hand.

JVM Tuning from the Trenches

This article is a follow-up to http://jojovedder.blogspot.com/2009/06/slow-weblogic-part-6-jvm-heap-analysis.html. Please read that one first for the basics on JVM heap, parameters and flags.

Also remember these tips have worked for the server settings and issues faced below, but blindly using these on your server will not produce the same results. You have to measure and then tune after measuring.

Problem:
Platform running Weblogic 8.1 on Sun V880 servers. Total RAM of 32 Gb on the machine.
2 Gb assigned to the managed server JVM heap. JDK 1.4

Initial settings:

-XX:+AggressiveHeap -Xms2048m -Xmx2048m  -XX:SurvivorRatio=32 -XX:MaxPermSize=128m 

But still there are 20 Full GCs per hour in peak times, before the server crashes.


Analysis

1. It was decided to reduce the SurvivorRatio to 4 and restart with some more flags.

The size of ONE Survivor Space is calculated as

SurvivorSpace = NewSize / (SurvivorRatio + 2)

Keeping SurvivorRatio as 32 means the Survivor spaces are too small for promoting stuff from Eden. Hence we reduce this to 4 which allows for larger Survivor spaces.

2. As per Sun Bug ID: 6218833, setting AggressiveHeap set before Heapsize (Xmx and Xms) can confuse the JVM. Revert the order to have -Xms and -Xmx to come before -XX:+AggressiveHeap or not use it

3. The application has 180+ EJBs with pools of beans. Hence set the -Dsun.rmi.dgc.client.gcInterval=3600000 (1 hour) instead of the default 60000 (1 min). More on this here: http://docs.sun.com/source/817-2180-10/pt_chap5.html

4. The site is restarted once a week at 4:30AM. The patterns stays normal for 2 days – and then degrades into full GC.

5. The Old space is pretty much full – at every minor collection – the Old space must be cleared up for promotion from Young to Old to take place.

6. Permanent space is pretty much full – keeps loading classes and classes ( could that be a problem – the difference between the number of JSP’s per Release?)
Hence we increased the PermSpace from 128M to 256M

7. Ensure we are running the server JVM by using the -server flag

8. Use OptimizeIt or similar profiling tool to see the memory usage and find code bottlenecks.


The settings now were

-server -Xms2048m -Xmx2048m  -XX:MaxNewSize=512m -XX:NewSize=512m -XX:SurvivorRatio=4 -XX:MaxPermSize=256m -Xincgc -XX:+DisableExplicitGC -XX:+AggressiveHeap -XX:-OmitStackTraceInFastThrow

This reduced the Full GCs to one a day.

Error Logs

At the time of the server going out of memory prior to a crash, the logs are filled with repeated errors (up to 100 repetitions) of this sort

java.lang.NullPointerException
 <<no stack trace available>>

Adding the -XX:-OmitStackTraceInFastThrow flag resolves this problem, the root cause of the NPE it self has to be tracked down but we do not have any longer the issue of huge recursive exception strings.

We could now see the stack trace as

java.lang.NullPointerException
 at java.util.StringTokenizer.(StringTokenizer.java:117)
 at java.util.StringTokenizer.(StringTokenizer.java:133)
 at jsp_servlet._framework._security.__login._jspService(login.jsp:294)
 at weblogic.servlet.jsp.JspBase.service(JspBase.java:27)
 at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1075)

This seems to be a Sun bug described here.

More on Weblogic Muxers

This is a follow-up to an earlier article which detailed the Weblogic Socket Muxer pattern in thread dumps.


Another ERROR pattern associated with Muxers is shown below

####<Jun 23, 2009 9:25:24 AM BST> <Error> <Socket> <d001> <managed1> <ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> 
 <BEA-000403> <IOException occurred on socket: Socket[addr=/10.23.22.1,port=1950,localport=61002]
 java.net.SocketException: Connection refused.
java.net.SocketException: Connection refused
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:672)
        at weblogic.socket.SocketMuxer.readReadySocket(SocketMuxer.java:648)
        at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:123)
        at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
        at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
        at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)

As written in an earlier post, each incoming request on a TCP socket uses one File Descriptor.

We monitored the File Descriptors available to this Weblogic server using

Here the Weblogic pid = 8359

/usr/proc/bin/pfiles 8359 | grep rlimit


1024

ls /proc/8359/fd | wc –l


140

The first command shows that 1024 FDs are allocated to this process, and the second shows how many are being used in real-time.

A plot of the second value during high load times shows the FDs go up to 900+ in use and hence a resolution used in this case is to increase the number of FDs.

A recommended value is to increase it to 8192.

Slow Weblogic Part 6 - JVM Heap Analysis using GCViewer

Basics


In an earlier article, I had listed the review of JVM memory parameters as one of the important checks for tuning the JEE server platform.

The basic primer for JDK 1.4 is at http://java.sun.com/docs/hotspot/gc1.4.2/


The key points you need to know are:

Total JVM Heap = Young + Tenured(also called Old)

Young = Eden + From (SS1) + To (SS2)

In the diagram below [taken from the Sun website], "From" and "To" are the names of the two Survivor Spaces (SS) within the "Young".

Perm Space (and code cache): stores JVM’s own stuff. This is outside the Heap you assign using Xms and Xmx. A good explanation of this is available here

The JVM Heap is at default initial 2Mb and max 64Mb (for JDK 1.4 on Solaris).
Default Perm Size is 16MB (for JDK 1.4 on Solaris)
The defaults change for each JDK and are different on each OS - so look up the values on the respective websites.

The ratios are as shown below

Now the object life cycle and garbage collection occurs like this:

1. Objects when created are always first allocated to Eden.
2. When Eden fills up, a fast but not comprehensive GC (minor collection) is run over the young generation only.
3. All surviving objects are moved from Eden into one Survivor Space.
4. In consequent minor collections, new objects move from Eden into the other Survivor Space, plus everything from the first Survivor Space (survivors from the previous minor collection) is also moved into the second Survivor Space. Thus one survivor should be empty at that time.
5. When objects in Survivor Space are old enough (or survivor fills up), they are moved to Tenured. By default the long-lived objects may be copied up to 31 times between the Survivor Spaces before they are finally promoted to the Old generation.
6. When tenured fills up, a Full GC collection is run that is comprehensive: the entire heap is analyzed, all objects that can be destroyed are killed and memory is reclaimed.

Note: the above lifecycle changes slightly when advanced options such as ConcurrentMarkSweep etc are enabled.

Look Closer

What do these values mean ?

A full list of options available at http://java.sun.com/docs/hotspot/VMOptions.html and http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp


The absolute basic ones are listed in the table below. Note: This is for JDK 1.4
Some of these have changed in JDK 1.6

-Xms1536m -Xmx1536m	These represent the total heap (minus Perm space). Xms is the Initial Heap, set to 1.5Gb in this case. Xmx is Max Heap. It is good practice to set Xms = Xmx The max heap is limited by the RAM available on the server
-XX:NewSize=512m	This specifies the initial size of the Young generation,set to 512Mbin this example. It is better to set this as a percentage of the Heap using -XX:NewRatio
-XX:MaxNewSize=512m	This specifies the maximum size of the Young generation,set to 512Mbin this example. It is better to set this as a percentage of the Heap using -XX:MaxNewRatio
-XX:PermSize=64m -XX:MaxPermSize=128m	These values are the Minimum and Maximum sizes of the permanent generation heap space. Optimally, set PermSize equal to MaxPermSize to avoid heap having to be adjusted when permanent area grows. As specified earlier, this area of memory is over and above the Total Heap set using Xms
-XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90	The New generation area is divided into three sub-areas: Eden, and two survivor spaces that are equal in size. Use the -XX:SurvivorRatio=X option to configure the ratio of the Eden/survivor space size. In the above example, setting it to 8 means the ratio of Eden:SS1:SS2 is 8:1:1. So for a NewSize of 512 Mb, the two SS will be 51 Mb each, and Eden will be 512 MINUS (51 + 51) = 410 Mb. TargetSurvivorRatio of 90 allows 90% of the survivor spaces to be occupied instead of the default 50%, allowing better utilization of the survivor space memory.
-XX:MaxTenuringThreshold=10	This switch determines how many times the objects are hopped between the Survivor spaces before getting promoted to the older generation. The default value is 31.
-XX:+DisableExplicitGC
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -Xloggc:/log/gc.log	These are GC specific settings asking for GC details and the log file name in which these details should be captured

If you have an appetite for more, read this http://java.sun.com/performance/reference/whitepapers/tuning.html

GCViewer

This link below explains how to download and use the GCViewer tool. This is quite a useful tool for viewing the number of GCs and Full GCs and how the JVMis behaving over time.

http://www.javaperformancetuning.com/tools/gcviewer/index.shtml

The most important things to look at in the GCViewer analysis are the

* Acc Pauses - Accumulated Pause Time (total time app was stopped for GC).Pauses are the times when an application appears unresponsive because garbage collection is occurring
* Total Time - Total Time the application runs.
* Throughput - Time the application runs and is not busy with GC. Greater than 99% is fantastic.Throughput is the percentage of total time not spent in garbage collection, considered over long periods of time.
* Footprint - Overall Memory Consumption - Ideally as low as possible. This is the working set of a process, measured in pages and cache lines. On systems with limited physical memory or many processes, footprint may dictate scalability. Thus this usually reflects the size of total Heap allocated via Xms and Xmx

This diagram is taken from the above site:



Tuning Example From the Trenches - Frequent GC due to Perm Space getting Full

JVM Parameters already set

java -server -Xms1024m -Xmx1024m -XX:MaxPermSize=340m -XX:NewSize=340m -XX:MaxNewSize=340m
-XX:SurvivorRatio=9 -XX:TargetSurvivorRatio=90 -XX:+UseParNewGC
-Xloggc:/wls_domains/gclog/jms.gc -XX:+PrintGCDetails -XX:+UseParNewGC
-XX:+PrintGCTimeStamps -XX:+DisableExplicitGC -XX:+PrintTenuringDistribution
-XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime
-XX:+JavaMonitorsInStackTrace -Dweblogic.system.BootIdentityFile=/wls_domains/xx/boot.properties
-Dweblogic.management.server=http://xx.xx.xx.xx:6000 -Dweblogic.Name=xxxxxx
-Dweblogic.ProductionModeEnabled=true
-Djava.security.policy=/opt/bea/wls/8.1sp4/weblogic81/server/lib/weblogic.policy
weblogic.Server

Full GC Pattern

The GC log shows

0.000: [Full GC 0.000: [Tenured: 0K->9794K(700416K), 0.8050952 secs] 134607K->9794K(1016960K), [Perm
: 20479K->20479K(20480K)], 0.8053527 secs]

First Full GC happens at 0:00 sec from start of server. Before and after '->' figures represent size of live objects before and after GC. Number in parenthesis indicates total available space. So from above numbers - Tenured was not full but Perm was almost full.

6579.013: [Full GC 6579.013: [Tenured: 9794K->18941K(700416K), 0.9677233 secs]
155600K->18941K(1016960K), [Perm : 24575K->24575K(24576K)], 0.9679896 secs]

Same thing happens here. The second Full GC took place at 6579.013 sec (1hr 49mins) from startup of server.
Again a Full GC is triggered, but the Tenured was not full. The Tenured is now 18.9 Mb out of 700 Mb - but it is seen the Perm Space has grown to 24.5 Mb and is not getting cleared.

9363.515: [Full GC 9363.516: [Tenured: 18941K->19463K(700416K), 0.6532332 secs] 36950K->19463K(1016960K), [Perm : 28672K->26462K(28672K)], 0.6536095 secs]

At the 3rd Full GC at 9363 seconds after server startup, the Perm space grew to 28.6 Mb and recovered marginally to 26.4 Mb.

Observing this over a long period of time, we concluded that at startup around 20MB is allocated to perm which with each Full GC keeps growing till 30MB and later shrinks back to 25MB and the cycle continues.

The pattern is highlighted below:

0.000: [Full GC 0.000: [Tenured: 0K->9794K(700416K), 0.8050952 secs] 134607K->9794K(1016960K), [Perm : 20479K->20479K(20480K)], 0.8053527 secs]
6579.013: [Full GC 6579.013: [Tenured: 9794K->18941K(700416K), 0.9677233 secs] 155600K->18941K(1016960K), [Perm : 24575K->24575K(24576K)], 0.9679896 secs]
9363.515: [Full GC 9363.516: [Tenured: 18941K->19463K(700416K), 0.6532332 secs] 36950K->19463K(1016960K), [Perm : 28672K->26462K(28672K)], 0.6536095 secs]
13483.233: [Full GC 13483.233: [Tenured: 19463K->16962K(700416K), 0.9783693 secs] 26678K->16962K(1016960K), [Perm : 30719K->21330K(30720K)], 0.9857390 secs]
17308.829: [Full GC 17308.830: [Tenured: 16962K->17312K(700416K), 1.0578872 secs] 88025K->17312K(1016960K), [Perm : 25600K->25600K(25600K)], 1.0581738 secs]
21237.810: [Full GC 21237.810: [Tenured: 17312K->17814K(700416K), 1.4728764 secs] 302290K->17814K(1016960K), [Perm : 29695K->26719K(29696K)], 1.4801234 secs]
30079.672: [Full GC 30079.672: [Tenured: 17814K->18676K(700416K), 1.0282446 secs] 83159K->18676K(1016960K), [Perm : 30975K->27564K(30976K)], 1.0349869 secs]

Solution:
Though the MaxPermSize=340m is provided, initial available Perm Size is getting full and JVM is invoking a Full GC to free up memory. The default initial PermSize is 16Mb and hence the Perm Space is resizing itself as the JVM grows.

Not sure if this behaviour is a bug but by adding an initial PermSize of 64K using this flag -XX:PermSize=64m resolved this issue of frequent Full GC.


UPDATE: Another examples published on http://jojovedder.blogspot.com/2009/07/jvm-tuning-from-trenches.html

Weblogic - Socket Muxers in Thread Dumps

What are these weblogic.socket.Muxer threads seen in thread dumps ?

Note: for a basic primer on taking thread dumps and analyzing them, see this earlier article

Socket Reader Threads accept the incoming request from the Listen Thread Queue and put it on the Execute Thread Queue.

In WL 8.1, there are 3 socket reader threads by default.
In WL 9 and 10, WebLogic allocates 33% of server threads to act as socket readers by default. This need not be changed usually.

One socket reader thread is usually in the poll function, while the others are available to process requests.
The polling thread is highlighted in the thread dump below.

"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x016b2148 nid=0x42 waiting for monitor entry [5997f000..5997fc28]
at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:91)
- waiting to lock <0x94846b40> (a java.lang.String)
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)

"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x00683c28 nid=0x41 waiting for monitor entry [59a7f000..59a7fc28]
at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:91)
- waiting to lock <0x94846b40> (a java.lang.String)
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)

"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x0079e5b0 nid=0x40 runnable [59b7f000..59b7fc28]
at weblogic.socket.PosixSocketMuxer.poll(Native Method)
at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:100)
- locked <0x94846b40> (a java.lang.String)
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)

In an earlier support case on Stuck Threads, we asked BEA:

Should we worry about the Weblogic.socket.Muxer threads which always show 2 threads waiting for lock and 3rd thread locking the same object?

The Muxer TD is attached. This shows same behaviour on all our Weblogic servers.

Full thread dump Java HotSpot(TM) Server VM (1.4.2_05-b04 mixed mode):

"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x0151
c588 nid=0x1b4 waiting for monitor entry [ad57f000..ad57fc28]
        at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:91)
        - waiting to lock <0xd9331760> (a java.lang.String)
        at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
        at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
        at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)

"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x0161
d608 nid=0x1b3 runnable [ad67f000..ad67fc28]
        at weblogic.socket.PosixSocketMuxer.poll(Native Method)
        at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:100)
        - locked <0xd9331760> (a java.lang.String)
        at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
        at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
        at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)

"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x01bb
6730 nid=0x1b2 waiting for monitor entry [ad77f000..ad77fc28]
        at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:91)
        - waiting to lock <0xd9331760> (a java.lang.String)
        at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
        at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
        at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)

The reply from BEA Support was that the above pattern of weblogic.socket.Muxer threads are not a cause of stuck threads.


Why do they mostly show as being Stuck in Samurai TD analyzer ?




As the image shows, when you analyze thread dumps using Samurai, the muxer threads are shown as being Stuck since they're all locked on the same object. This is probably treated as a deadlock condition.

"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'"
- waiting to lock <0xd9b61098> (a java.lang.String)

"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'"
- waiting to lock <0xd9b61098> (a java.lang.String)

"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'"
- locked <0xd9b61098> (a java.lang.String)

But you will see the same in any Thread dump even on a development instance with no requests.
The locks mentioned do show up as red in Samurai - but they aren't deadlocks just regular locks.

A thread gains an exclusive lock on an object to perform some action, then frees it allowing the next thread to gain access.

Additionally, if you look at the thread dumps over time, you'll see that these specific locks are not always present - they are moving between the threads which is indicative of their transitory nature.



I want to know more details on Muxers


The socket Muxer manages the server’s existing socket connections.
It first determines which sockets have incoming requests waiting to be processed. It then reads enough data to determine the protocol and dispatches the socket to an appropriate runtime layer based on the protocol.
In the runtime layer, the socket muxer threads determine which execute thread queue to be used and delegates the request accordingly.

From the documentation on http://edocs.bea.com/wls/docs100/perform/WLSTuning.html#wp1152246 ,
Weblogic has two versions of the socket muxer, one is the Java version and the other uses a native library which makes better use of operating system calls. The Enable Native IO checkbox on the server’s configuration settings tells the server which version to use. This is ON by default for most platforms.

Native muxers provide superior scalability because they implement a non-blocking thread model. When a native muxer is used, the server creates a fixed number of threads dedicated to reading incoming requests. Oracle recommends using the default setting of true for the Enable Native IO parameter which allows the server to automatically select the appropriate muxer to use.

You must ensure that to use Native I/O, the native library must be present in the server’s shared library path . This is set up with the default scripts.
When the server does not find the native library, it throws an error
java.lang.UnsatisfiedLinkError: no muxer in java.library.path
and then loads the Java version of the muxer.

Confirm the LD library path is okay and pointing to the Solaris LD path. Check the startup log when starting a managed server. What is the value of java.library.path?
This is where the JVM actually get's the library from.

http://m-button.blogspot.com/2008/08/how-does-weblogic-handle-socket-muxers.html has a good example of how to identify Native vs Java muxer in a thread dump.

The Thread Dump I’ve used in my examples above uses the Native muxer (weblogic.socket.PosixSocketMuxer) on Solaris.

Solaris has another Native muxer called the weblogic.socket.DevPollSocketMuxer
An example TD using this muxer is shown below.

"ExecuteThread: '4' for queue: 'weblogic.socket.Muxer'" waiting for lock java.lang.String@4edf4f BLOCKED
weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:95)
weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

"ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" RUNNABLE native
weblogic.socket.DevPollSocketMuxer.doPoll(Native Method)
weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:96)
weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)

weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)
"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" waiting for lock java.lang.String@4edf4f BLOCKED
weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:95)
weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" waiting for lock java.lang.String@4edf4f BLOCKED
weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:95)
weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" waiting for lock java.lang.String@4edf4f BLOCKED
weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:95)
weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

To change the number of Muxers from the default, follow the instructions given at http://e-docs.bea.com/wls/docs92/ConsoleHelp/taskhelp/tuning/TuningSocketReaders.html




See http://jojovedder.blogspot.com/2009/07/more-on-weblogic-muxers.htmlfor an update on Muxers

Additionally on Oracle JRockit JVMs - there are some information in the thread dumps which point out the same problem in a different manner.

After the normal stack dumps, BEA JRockit performs a deadlock detection. This is done by finding "lock chains" in the Java application. If a lock chain is found to be circular, the application is considered caught in a deadlock.

A detailed explanation of the 3 types of lock chains in JRockit is given here


What is relevant for us is the example of Muxers which are shown as:


Blocked lock chains
===================
Chain 2:
"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=129 idx=0x218 tid=4079 waiting for java/lang/String@0x37804000 held by:
"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=127 idx=0x210 tid=4077 in chain 1

Open lock chains
================
Chain 1:
"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" id=128 idx=0x214 tid=4078 waiting for java/lang/String@0x37804000 held by:
"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=127 idx=0x210 tid=4077 (active)


As per the explanation, the Open lock chain depicts Thread 1 waiting for Thread 0. This is not a deadlock, only a straight dependency.

Since Thread 0 is already part of the Open lock chain, the fact that Thread 2 is also waiting on the same Thread 0 is treated as a "Blocked lock chain".
In this case this is not a problem.

Update 15th Feb 2011

I'm glad this blog entry on weblogic muxers has made it onto the Oracle forums with a mention from James Bayer.