AWS deployment options - Part 3 - mod_proxy with autoscaling

Due to the limitation with mod_jk for autoscaling configurations, we took a look at mod_proxy

Instead of using AJP (mod_jk) protocol we can use the HTTP protocol and with this approach we do not use Apache for the actual load balancing or session stickiness. Instead all requests from the Apache server are forwarded to the application tier ELB (say it runs at HTTP port 8080) and the ELB does load balancing as well as maintains session stickiness.

The Apache httpd.conf configuration for the same is mentioned below.

LoadModule proxy_module       modules/mod_proxy.so

LoadModule proxy_http_module  modules/mod_proxy_http.so

ProxyPass /img !

ProxyPass /images !
ProxyPass /css !
ProxyPass /media !

ProxyPass / http://internal-RCCL-AppTier-ELB-722360436.us-east-1.elb.amazonaws.com:8080/

ProxyPassReverse / http://internal-RCCL-AppTier-ELB-722360436.us-east-1.elb.amazonaws.com:8080/

§Pros:

–ELB is able to maintain sticky sessions using it’s own cookie named AWSELB

§Cons:

–When we need to run multiple Tomcat JVMs on the same instance, this approach fails. For eg: If we have 6 Tomcats distributed as Server 1 on ports 8080, 8085, 8090 and Server 2 on ports 8080, 8085, 8090. In this case, on Apache we need to switch to mod_proxy_balancer and Apache will load balance across 3 ELBs running on 8080, 8085 and 8090 respectively.

–In such a scenario, session stickyness fails again as it again requires the same combination of unique node names (jvmRoutes) for each Tomcat instances and these need to be pre-defined on Apache as well. So we face the same issue on mod_jk See http://stackoverflow.com/questions/9393163/apache-tomcat-problems-with-sticky-sessions-and-load-balancing

 So if you need a single Tomcat per JVM then this is the best solution.

For multiple JVMs there is an alternate recommendation given to us by Amazon support, and that is to use one ELB per port and set up the tomcat tier with multiple ELBs as shown below.

 This will work provided you allocate the architecture and resources in the right manner.

 

AWS deployment options - Part 2 - mod_jk with autoscaling

Part 1 gave the overall deployment picture.

 

If the application requires session stickiness at the Tomcat tier, i.e. if some user details are stored in the HTTP Session and you don't enable session replication across Tomcats. The solution is that the Elastic Load Balancer (ELB) maintains the sticky session using it’s own cookie called AWSELB. For ELB sticky session configuration see http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.managing.elb.html

In case there is an external ELB (Apache tier) with cookies stickiness set, it will over-ride the internal ELB (Tomcat tier). The stickiness cookie is always named AWSELB, hence the external ELB will overwrite the cookie from the internal ELB. One option here is to set the Tomcat tier ELB with "Application Generated Cookie Stickiness" with value JSESSIONID so it will follow the Application cookie and maintain sticky sessions.

 

In case sticky session is not enabled on the application tier, and the user is forwarded to a Tomcat server where the session is not present, the user’s session information will be lost

There are various approaches available to maintain session stickiness, and these have been discussed below.

 

Option 1: Use mod_jk to get Apache to direct to the correct Tomcat

Option 2: Use mod_proxy

Option 3: No session stickiness, keep it stateless (which is what Amazon recommends)

The mod_jk implementation looks like this

 

 

The Apache web server can maintain session stickiness along with load balancing i.e. The requests from the Apache server to the Tomcat server are load balanced along with maintaining session stickiness, by Apache server itself .(The application ELB is bypassed).

In this approach the Apache server needs to know the list of all possible Tomcat IPs.

When we set up autoscaling, Tomcat can start up new instances on any IP within the subnet. That's the way AWS works.

Hence this is not feasible as when we restart the AWS servers the IPs might change.

 

 

 

 Mod JK Configuration

The file “httpd.conf” at the location “/www/Apache/conf” needs to have the following entries. These entries indicate that the files from folders “img”, “images”, ”css” and “media” will be served locally where as rest of the requests will be forwarded to the load balancer “lb”.

LoadModule jk_module modules/mod_jk-1.2.30-httpd-2.2.X.so JkWorkersFile conf/workers.properties JkLogFile logs/mod_jk.log JkLogLevel error JkShmFile logs/mod_jk.shm   JKMount /status jkstatus   JkAutoAlias /apps/Tomcat/apache-tomcat-7.0.42/webapps   JkMount  /* lb JkUnMount /img/* lb JkUnMount /images/* lb JkUnMount /css/* lb JkUnMount /media/* lb

 

 

 

 

 

 

 

 

 

 

 

 

Also the file “worker.properties” will also need to be modified as indicated below.

worker.list=lb,jkstatus   #node 1 worker.node1.type=ajp13 worker.node1.port=8009 worker.node1.host=172.23.0.36 worker.node1.lbfactor=1 worker.node1.cachesize=10   #node 2 worker.node2.type=ajp13 worker.node2.port=8009 worker.node2.host=172.23.0.38 worker.node2.lbfactor=1 worker.node2.cachesize=10   #node 3 worker.node3.type=ajp13 worker.node3.port=8009 worker.node3.host=172.23.1.36 worker.node3.lbfactor=1 worker.node3.cachesize=10

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 This maps to the Tomcat server.xml for each JVM

 

 

§Pros:

–Has very good support for failure detection and switchover to alternate Tomcat server.

–Use ELB for health checks and not for load balancing

§Cons:

–The session stickyness is using cookies like this jsessionid=AB327CD87667461357C95742BB9974B1.node1 which is maintained via configuration in Apache workers.properties. In this all Tomcat IP addresses and ports must be mentioned along with a logical name for each Tomcat (node1 in this case)
That logical name must be the same as entered in each individual Tomcat server.xml as it’s “jvmRoute” as well. Thus we will have Tomcat node1, node2 .. nodeN

–This affects autoscaling because Tomcats will be a new instance autoscaled on any available IP (from a set of IPs) and it’s not possible to hardcode a “nodeX” in Tomcat when it is launched via autoscaling

For this reason we next take a look at mod_proxy

 

 

 

Amazon AWS deployment options for Apache web server to Tomcat connectivity

The cloud proposition is attractive because of the ability to scale up or down to meet workload demand almost instantaneously and to utilize efficiencies in the services delivery model. The rationale of using and paying for only what you need, when you need it is a very compelling business justification.

 

This post and the next few ones describes the various options and features available when a web application is deployed on the AWS cloud infrastructure using Apache HTTP Web Server and Apache Tomcat application server. This document deals more with options available to maintain sticky sessions and application persistence.

 

This is based on a recent project where we are deploying with Apache 2.4 and Tomcat 7.0.x

 

First some basic terms

 

 AWS Glossary

 

·                     Amazon EC2 is hosted in multiple locations world-wide. These locations are composed of Regions and Zones. Each region is a separate geographic area.
Each region has multiple, isolated locations known as Availability Zones. This supports high availability of the applications and workloads can be balanced across both availability zones.

·                     To maintain a high-level of security the different tiers such as Web, App and Database are divided into separate subnets and "Security Groups", which are roughly analogous to firewalls.  This is done through Amazon VPC, a logically isolated section of the AWS Cloud where customers can launch AWS resources into a virtual network which they define. Capability to control inbound and outbound access to and from individual subnets using network access control lists, i.e. stateless firewall.  

·                     Load balancer (Elastic Load Balancer)

Elastic Load Balancing automatically distributes incoming application traffic across multiple Amazon EC2 instances. Elastic Load Balancing also detects unhealthy instances and automatically reroutes traffic to healthy instances until the unhealthy instances have been restored. Elastic Load Balancing can also be enabled within a single or multiple Availability Zones for more consistent application performance.

ELB also maintains Sticky sessions in case the application requires it.

 

Logical View

 

 

 

The application is hosted across two Zones within a region.

There is a Web tier ELB to load balance across the Apache servers as well as an Application tier ELB to load balance requests across the Tomcat servers. Cross-zone load balancing is optional and can be configured as required.

The requests from the Apache server to the Tomcat server can also be load balanced by Apache server itself (i.e. the application tier ELB is bypassed and not used). But this is not a preferred approach for AWS.

Hence both the application as well as the web tier ELB’s are used for load balancing.

In the next post I will explain how to use either mod_jk or mod_proxy to achieve the load balancing plus sticky sessions.

 

 

Now Playing - March 2014

Music albums that I've liked over the last few months:

Radical Face - love this album The Family Tree: The Branches - reminds me of Bon Iver and The Verve
The National - Trouble Will Find Me
Of Monsters and Men
The Secret Life of Walter Mitty - OST - Jose Gonzalez has a superb song titled Step Outside
Pearl Jam - Lightning Bolt

Aside on Bon Iver's Holocene - which was part of the OST for Cameron Crowe's "We Bought a Zoo" - the lyrics and tune remind you of vast open white wilderness (ignore the geological meaning) - which reminds me of what a Vanilla Sky would look like - which is another of Cameron Crowe's films.

Solution for Samsung Galaxy Hanging/Freezing Screen - Samsung Galaxy Core Duos GT-I8262

My Galaxy Core hangs all the time. I uninstalled all downloaded apps and also I did a factory reset, but that also didn’t help.

Some googling led me onto this post on XDA developers. If you want more info (and know more about the risks), go to the XDA forum and read the whole thing.

Temporary Fix

So the steps are install Dummy File Generator (DFG) on your phone. This app can populate your entire internal memory and SD-card memory with dummy data. I completely filled the internal memory with that tool. After that go back to the tool and choose "Delete" of all the dummy files.

You will get a warning message that memory is nearly full. Whatever you do - do not reboot in between this, because the phone won't reboot after that.

If the DFG hangs when trying to delete dummy files, then go to Application Manager > Dummy File Generator > Clear Data.

After this there will still be the odd freeze but not as severe as before (may be once a day).

More Reading:
http://androidforums.com/samsung-galaxy-s3/749952-solved-moment-s3-apps-crash-phone-freezes-everyday-factory-reset-already.html
http://forums.androidcentral.com/samsung-galaxy-siii-i9300/269096-s3-hanging-freezing-problem-solved.html

Better Fix

I took the phone to the Samsung mobile centre, they told me the problem actually is introduced after you do a software upgrade directly over wi-fi or network. They recommend downgrading to Factory version and they do the software upgrade for you and in 90% of the cases, this fixes the hanging problem.