Some info on Weblogic clustering

Recently this question regarding Weblogic clustering came up on Stackoverflow.

I put some time into researching the answer I provided, and thought of putting this up on the blog as well.

Question from user Russell

I've read http://download.oracle.com/docs/cd/E11035_01/wls100/cluster/overview.html and searched this topic on the internet but still had a hard time understanding some of weblogic's clustering concepts.

Can anybody confirm/correct my understandings below?

•a cluster contains one or more logical servers which can reside on one or many physical servers
•when deploying a j2ee app to a cluster, it is tied to one server in that cluster
•external users of the deployed app aren't aware of clustering
•the log file of that app is located on the server it's deployed
•if the server hosting the app fails, it's okay because the app is in a cluster and another server will pick up the work?
•if the server hosting the app fails, what happens to logging?
Maybe I got the whole concept wrong. Could anybody point me into the correct directions?

Answer

It's good to understand the concept of Domain first.

The Domain is the parent of a Cluster. It contains typically one Admin and one or more Managed servers.
Now the Cluster is a grouping of some or all of these managed servers within the domain.

This diagram here should help understanding the relation between Domains and Clusters

Once you configure a Domain and a Cluster yourself on a development environment, you'll get to know more about it.

Now here are the answers to your specific questions

•a cluster contains one or more
logical servers which can reside on
one or many physical servers

True. But let's clarify what you mean by 'logical' servers.
In the Cluster you typically have two or more Managed servers. These servers run in their own JVMs and can be started independently and serve requests independently. Each server will have a unique IP:port address, and it can be directly accessed from the browser. But these server instances can reside over multiple physical servers.

•when deploying a j2ee app to a
cluster, it is tied to one server in
that cluster

No it is not tied to one server. When you deploy a J2EE app to the Cluster, it will get deployed in turn to each server in that cluster. The JNDI is cluster-wide and each server maintains a local copy of the JNDI.

You can look up the object (say an EJB) via JNDI on the Cluster or on the individual server. Also see what types of Objects can be clustered.

•external users of the deployed app
aren't aware of clustering

True.

But in this case you should have an Apache web server or a load balancer or DNS server which takes the request from the browser, and internally maps it to one of the servers in the cluster. If you dont have any of these, you would have to define the cluster address as a DNS name or IP address for the client. See the section "Avoiding Listen Address Problems " on http://download.oracle.com/docs/cd/E13222_01/wls/docs103/cluster/setup.html#wp682940

•the log file of that app is located
on the server it's deployed

True, one weblogic log per server.

•if the server hosting the app fails,
it's okay because the app is in a
cluster and another server will pick
up the work?

Not by default, you have to configure it for failover and replication. This is a huge topic which needs separate reading

•if the server hosting the app fails,
what happens to logging?

Logging stops. You'll see some shutdown or heartbeat errors in the log, or outofmemory or whatever reason for failure. you'll have to restart the server - and logging continues in a new file (depending on your logging settings)

My original answer is here on stackoverflow for those interested.

Weblogic JDK 1.6 Error Could not reserve enough space for object heap

Weblogic 10 running on HP-UX throws this error on startup when we try to give 3 GB Xms and Xmx.

-Dweblogic.system.BootIdentityFile=/portalapp/bea/user_projects/domains/MSD4/servers/MS4/data/nodemanager/boot.properties -Dweblogic.nodemanager.ServiceEnabled=true -Dweblogic.security.SSL.ignoreHostnameVerification=false -Dweblogic.ReverseDNSAllowed=false -server -Xms3g -Xmx3g -Xmn1024m -XX:+AggressiveHeap -XX:+UseParallelGC -XX:ParallelGCThreads=20 -XX:PermSize=1024m -XX:MaxPermSize=1024m -XX:MaxTenuringThreshold=3 -XX:LargePageSizeInBytes=4m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:SurvivorRatio=8 -XX:-UseAdaptiveSizePolicy -Xloggc:/portal_logs/gclog/MS4_gclog.log weblogic.Server >

<Feb 17, 2010 10:56:06 AM> <Info> <NodeManager> <Working directory is "/portalapp/bea/user_projects/domains/MSD4">

<Feb 17, 2010 10:56:06 AM> <Info> <NodeManager> <Server output log file is "/portalapp/bea/user_projects/domains/MSD4/servers/MS4/logs/MS4.out">
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.
<Feb 17, 2010 10:56:06 AM> <Debug> <NodeManager> <Waiting for the process to die: 13663>
<Feb 17, 2010 10:56:06 AM> <Info> <NodeManager> <Server failed during startup so will not be restarted>

Additional information:

os.name = HP-UX
os.version = B.11.31

java.version = 1.6.0.04
java.vm.info = mixed mode
java.vm.name = Java HotSpot(TM) Server VM
java.vm.specification.name = Java Virtual Machine Specification
java.vm.specification.vendor = Sun Microsystems Inc.
java.vm.specification.version = 1.0
java.vm.vendor = "Hewlett-Packard Company"
java.vm.version = 11.3-b02-jre1.6.0.04-rc2
java.runtime.version = 1.6.0.04-jinteg_28_apr_2009_04_46-b00


Analysis:


Though we are using a 64-bit server which has 16 GB RAM, we are unable to assign more than 2 Gb to the Weblogic heap.

This value in the startup log shows that the JVM is a 32-bit one.

sun.arch.data.model = 32


In a 32-bit JVM – the process cannot address memory higher than 4 GB in all. But as per Sun site, in reality we cannot get more than 2G.

As per this URL: http://java.sun.com/docs/hotspot/HotSpotFAQ.html#gc_heap_32bit

Why can't I get a larger heap with the 32-bit JVM?

The maximum theoretical heap limit for the 32-bit JVM is 4G. Due to various additional constraints such as available swap, kernel address space usage, memory fragmentation, and VM overhead, in practice the limit can be much lower. On most modern 32-bit Windows systems the maximum heap size will range from 1.4G to 1.6G. On 32-bit Solaris kernels the address space is limited to 2G.

So we tuned the heap to 2 GB for now. i.e. set Xmx and Xms to 2G and that solved it.

More tuning tips on slow Weblogic 10

In the earlier parts of this article I described some technical tuning tips specific to a slow-performing Weblogic JEE server.

Another earlier post has looked at how to set the JSP check seconds values.

This is a follow-up article specifically for tuning the similar settings for Weblogic 10 - since the syntax has changed.


In weblogic.xml, ensure the following are set as below:

<wls:session-descriptor>
<wls:encode-session-id-in-query-params>true</wls:encode-session-id-in-query-params>
<wls:cookie-comment>mysessioncookie</wls:cookie-comment>
<wls:cookie-max-age-secs>-1</wls:cookie-max-age-secs>
<wls:cookie-name>JSESSIONID</wls:cookie-name>
<wls:cookie-path>/</wls:cookie-path>
<wls:cookies-enabled>true</wls:cookies-enabled>
<wls:timeout-secs>1200</timeout-secs>
<wls:cookie-domain>@MY_COOKIE_DOMAIN@</wls:cookie-domain>
</wls:session-descriptor>



Assuming that no JSPs are being hot deployed or dropped directly into the production server

<wls:jsp-descriptor>
<wls:keepgenerated>true</wls:keepgenerated>
<wls:precompile>true</wls:precompile>
<wls:page-check-seconds>-1</wls:page-check-seconds>
</wls:jsp-descriptor>


and this one

<wls:container-descriptor>
<wls:servlet-reload-check-secs>-1</wls:servlet-reload-check-secs>
</wls:container-descriptor>


Ensure all of these are inserted at the right place within the weblogic.xml to avoid any start up error messages

Check against the reference here http://download.oracle.com/docs/cd/E12840_01/wls/docs103/webapp/weblogic_xml.html

Postcards from New Zealand

I spent 2 months on a client assignment in New Zealand last year. The summer was a great time to be there and here are some pictures.



Auckland skyline




Looking up at the SkyCity Tower Auckland





Cathedral Cove in the North Island







View of Auckland from Waiheke Island







The Cruise on Milford Sound in the South Island. The view ahead is the Trans Tasman Sea.





The seals at Milford Sound




Birds at Queenstown




Lake Pukaki on the way to Mount Cook




View of Mount Cook from Lake Tekapo



The Church of Good Shephed on the shores of Lake Tekapo




View of Lake Tekapo from Mount John Observatory




Two men playing chess at Christchurch City Centre. Let me know if it's you :)