Thursday, 7 December 2017

Practical Learnings on Microservices

In a recent project, we delivered over 40 Microservices deployed on 4 Tomcat servers each in Production.

Below are some notes on real-life learnings.

Designing the Microservices:

Business focus
Small services with CRUD operations on single business function or domain 
•Use Lightweight REST based communication (client-to-service and service-to-service)

        Keep Loosely Coupled
       •Ensure Services are Stateless
•Appropriate design patterns such as Aggregator, Proxy & Branch Patterns are commonly used
•In unavoidable cases - there will be Distributed Transactions and need to design for them
•Must design for Failure – e.g. Delays, Errors or Unavailability of another service or 3rd party system.
•Provide default functionality in case of failures from a service
•Rely on Input Validation - (client-to-service and service-to-service)
•Centralized Logging and Monitoring is a must across distributed microservices

Log events for timeouts and shut downs
Logging to include the level, hostname (instance name), message
Log events can be used for capacity planning and scaling e.g. which services need higher instances
Business data related metrics such as no. of Bookings, Time taken to fill out the form 
•Log events which can be used for capacity planning and scaling
•Use Testing tools for integration of services
•Quick feedback on check-ins and failures in the CI/CD pipeline

§Independent Development teams
Business domain-driven design
Responsive to business changes
§Independent Deployments
Minimal impact and regression testing for small fixes
Able to quickly take down and redeploy one service without impact to the overall system or other functional services.
§Independent Scalability
Slots well into on-demand hosting and cloud services for scalability 

§Reusability for other Enterprise projects


§Higher Complexity for developers initially
§Deal with error handling, timeouts, retry mechanisms
§Service versioning can get confusing and needs to be managed.
§Managing Distributed Transactions requires Design rework.
§Requires increased number of VMs to support environments adding to infrastructure costs
§Need clear deployment strategy at the start to identify which services will be deployed on which VMs
§Increased overhead for Configuration management and Rundeck/UDeploy deployments during project life cycle and in production as a result of 45+ separate code builds and deployments

§Across environments, there are 180+ Tomcat Servers in Production and 360+  Servers in lower environments

Monday, 26 May 2014

AWS deployment options - Part 3 - mod_proxy with autoscaling

Due to the limitation with mod_jk for autoscaling configurations, we took a look at mod_proxy

Instead of using AJP (mod_jk) protocol we can use the HTTP protocol and with this approach we do not use Apache for the actual load balancing or session stickiness. Instead all requests from the Apache server are forwarded to the application tier ELB (say it runs at HTTP port 8080) and the ELB does load balancing as well as maintains session stickiness.

The Apache httpd.conf configuration for the same is mentioned below.

LoadModule proxy_module       modules/
LoadModule proxy_http_module  modules/

ProxyPass /img !
ProxyPass /images !
ProxyPass /css !
ProxyPass /media !

ProxyPass /
ProxyPassReverse /


ELB is able to maintain sticky sessions using it’s own cookie named AWSELB


When we need to run multiple Tomcat JVMs on the same instance, this approach fails. For eg: If we have 6 Tomcats distributed as Server 1 on ports 8080, 8085, 8090 and Server 2 on ports 8080, 8085, 8090. In this case, on Apache we need to switch to mod_proxy_balancer and Apache will load balance across 3 ELBs running on 8080, 8085 and 8090 respectively.

In such a scenario, session stickyness fails again as it again requires the same combination of unique node names (jvmRoutes) for each Tomcat instances and these need to be pre-defined on Apache as well. So we face the same issue on mod_jk See
So if you need a single Tomcat per JVM then this is the best solution.

For multiple JVMs there is an alternate recommendation given to us by Amazon support, and that is to use one ELB per port and set up the tomcat tier with multiple ELBs as shown below.

This will work provided you allocate the architecture and resources in the right manner.


Friday, 23 May 2014

AWS deployment options - Part 2 - mod_jk with autoscaling

If the application requires session stickiness at the Tomcat tier, i.e. if some user details are stored in the HTTP Session and you don't enable session replication across Tomcats. The solution is that the Elastic Load Balancer (ELB) maintains the sticky session using it’s own cookie called AWSELB. For ELB sticky session configuration see
In case there is an external ELB (Apache tier) with cookies stickiness set, it will over-ride the internal ELB (Tomcat tier). The stickiness cookie is always named AWSELB, hence the external ELB will overwrite the cookie from the internal ELB. One option here is to set the Tomcat tier ELB with "Application Generated Cookie Stickiness" with value JSESSIONID so it will follow the Application cookie and maintain sticky sessions.

In case sticky session is not enabled on the application tier, and the user is forwarded to a Tomcat server where the session is not present, the user’s session information will be lost

There are various approaches available to maintain session stickiness, and these have been discussed below.
Option 1: Use mod_jk to get Apache to direct to the correct Tomcat

Option 2: Use mod_proxy

Option 3: No session stickiness, keep it stateless (which is what Amazon recommends)

The mod_jk implementation looks like this
The Apache web server can maintain session stickiness along with load balancing i.e. The requests from the Apache server to the Tomcat server are load balanced along with maintaining session stickiness, by Apache server itself .(The application ELB is bypassed).
In this approach the Apache server needs to know the list of all possible Tomcat IPs.
When we set up autoscaling, Tomcat can start up new instances on any IP within the subnet. That's the way AWS works.
Hence this is not feasible as when we restart the AWS servers the IPs might change.

 Mod JK Configuration

The file “httpd.conf” at the location “/www/Apache/conf” needs to have the following entries. These entries indicate that the files from folders “img”, “images”, ”css” and “media” will be served locally where as rest of the requests will be forwarded to the load balancer “lb”.

LoadModule jk_module modules/
JkWorkersFile conf/
JkLogFile logs/mod_jk.log
JkLogLevel error
JkShmFile logs/mod_jk.shm
JKMount /status jkstatus
JkAutoAlias /apps/Tomcat/apache-tomcat-7.0.42/webapps
JkMount  /* lb
JkUnMount /img/* lb
JkUnMount /images/* lb
JkUnMount /css/* lb
JkUnMount /media/* lb


Also the file “” will also need to be modified as indicated below.

#node 1
#node 2
#node 3


 This maps to the Tomcat server.xml for each JVM

Has very good support for failure detection and switchover to alternate Tomcat server.
Use ELB for health checks and not for load balancing
The session stickyness is using cookies like this jsessionid=AB327CD87667461357C95742BB9974B1.node1 which is maintained via configuration in Apache In this all Tomcat IP addresses and ports must be mentioned along with a logical name for each Tomcat (node1 in this case)
That logical name
must be the same as entered in each individual Tomcat server.xml as it’s “jvmRouteas well. Thus we will have Tomcat node1, node2 .. nodeN
This affects autoscaling because Tomcats will be a new instance autoscaled on any available IP (from a set of IPs) and it’s not possible to hardcode a “nodeX” in Tomcat when it is launched via autoscaling
For this reason we next take a look at mod_proxy


Thursday, 22 May 2014

Amazon AWS deployment options for Apache web server to Tomcat connectivity

The cloud proposition is attractive because of the ability to scale up or down to meet workload demand almost instantaneously and to utilize efficiencies in the services delivery model. The rationale of using and paying for only what you need, when you need it is a very compelling business justification.


This post and the next few ones describes the various options and features available when a web application is deployed on the AWS cloud infrastructure using Apache HTTP Web Server and Apache Tomcat application server. This document deals more with options available to maintain sticky sessions and application persistence.
This is based on a recent project where we are deploying with Apache 2.4 and Tomcat 7.0.x
First some basic terms

 AWS Glossary

·                     Amazon EC2 is hosted in multiple locations world-wide. These locations are composed of Regions and Zones. Each region is a separate geographic area.
Each region has multiple, isolated locations known as Availability Zones. This supports high availability of the applications and workloads can be balanced across both availability zones.
·                     To maintain a high-level of security the different tiers such as Web, App and Database are divided into separate subnets and "Security Groups", which are roughly analogous to firewalls.  This is done through Amazon VPC, a logically isolated section of the AWS Cloud where customers can launch AWS resources into a virtual network which they define. Capability to control inbound and outbound access to and from individual subnets using network access control lists, i.e. stateless firewall.  
·                     Load balancer (Elastic Load Balancer)
Elastic Load Balancing automatically distributes incoming application traffic across multiple Amazon EC2 instances. Elastic Load Balancing also detects unhealthy instances and automatically reroutes traffic to healthy instances until the unhealthy instances have been restored. Elastic Load Balancing can also be enabled within a single or multiple Availability Zones for more consistent application performance.
ELB also maintains Sticky sessions in case the application requires it.

Logical View

The application is hosted across two Zones within a region.
There is a Web tier ELB to load balance across the Apache servers as well as an Application tier ELB to load balance requests across the Tomcat servers. Cross-zone load balancing is optional and can be configured as required.
The requests from the Apache server to the Tomcat server can also be load balanced by Apache server itself (i.e. the application tier ELB is bypassed and not used). But this is not a preferred approach for AWS.
Hence both the application as well as the web tier ELB’s are used for load balancing.
In the next post I will explain how to use either mod_jk or mod_proxy to achieve the load balancing plus sticky sessions.


Saturday, 15 March 2014

Now Playing - March 2014

Music albums that I've liked over the last few months:

Radical Face - love this album The Family Tree: The Branches - reminds me of Bon Iver and The Verve
The National - Trouble Will Find Me
Of Monsters and Men
The Secret Life of Walter Mitty - OST - Jose Gonzalez has a superb song titled Step Outside
Pearl Jam - Lightning Bolt

Aside on Bon Iver's Holocene - which was part of the OST for Cameron Crowe's "We Bought a Zoo" - the lyrics and tune remind you of vast open white wilderness (ignore the geological meaning) - which reminds me of what a Vanilla Sky would look like - which is another of Cameron Crowe's films.

Wednesday, 15 January 2014

Solution for Samsung Galaxy Hanging/Freezing Screen - Samsung Galaxy Core Duos GT-I8262

My Galaxy Core hangs all the time. I uninstalled all downloaded apps and also I did a factory reset, but that also didn’t help.

Some googling led me onto this post on XDA developers. If you want more info (and know more about the risks), go to the XDA forum and read the whole thing.

Temporary Fix

So the steps are install Dummy File Generator (DFG) on your phone. This app can populate your entire internal memory and SD-card memory with dummy data. I completely filled the internal memory with that tool. After that go back to the tool and choose "Delete" of all the dummy files.

You will get a warning message that memory is nearly full. Whatever you do - do not reboot in between this, because the phone won't reboot after that.

If the DFG hangs when trying to delete dummy files, then go to Application Manager > Dummy File Generator > Clear Data.

After this there will still be the odd freeze but not as severe as before (may be once a day).

More Reading:

Better Fix

I took the phone to the Samsung mobile centre, they told me the problem actually is introduced after you do a software upgrade directly over wi-fi or network. They recommend downgrading to Factory version and they do the software upgrade for you and in 90% of the cases, this fixes the hanging problem.

Thursday, 13 June 2013

Maven build - java.lang.IllegalArgumentException: The argument does not represent an annotation type

When you run a Maven build and get this error

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.
0:compile (default-compile) on project celebrity-cda-web: Fatal error compiling:
 java.lang.IllegalArgumentException: The argument does not represent an annotati
on type: XStreamImplicit -> [Help 1]

In this case Maven cannot find the class XStreamImplicit

The problem is that a <dependency> which was earlier defined in the pom.xml has now been deleted or the Maven repository is not defined correctly for that dependency.