JVM Heap Sizing - 32 vs 64 bit

Continuing with my earlier discussion related to JVM memory management one shortcut adopted by many desperadoes to fix heap thrashes is to increase the max heap space. However, some may not have this luxury as they may be running on 32 bit OS. Others, may be reluctant to increase beyond 1GB thinking it may negatively affect JVM performance. These misconceptions are of the past, with Java5 and Java6, garbage collector which drives JVM memory management can handle up to 1.7GB with 100% efficiency on a Win32 machine (3.2GB on AIX 32-bit) and on a a 64-bit machine heap max is virtually unlimited, albeit with an efficiency of 50-70%. The key to note here is drop in efficiency rate of heap utilization, your JVM will now need 64 bit pointers to access java object reference. For example, an app that runs on 32 bit mode with 3GB heap space will now need 5-6GB heap space when migrated to 64 bit platform, so as to maintain its previous efficiency level.

Pointer Size	Space	Max Heap	Efficiency
32 bits	4GB	1.7GB (Win)	100%
		3.2GB(AIX)
64 bits	unlimited	unlimited	50-70%
64 bits compressed	4 to 32 GB	4 to 32 GB	~70-100%

With the introduction of Java6 (in particular Websphere 7.x), 64-bit compressed technology has been introduced where 64 bit JVM's can use 32 bit heap offset to reference heap data. This offers larger memory space than 32 bit space and reduces memory footprint on 64 bit, increased throughput and more efficient heap utilization. Compression is disabled by default, but can be enabled using -Xcompressedrefs switch.

Garbage Collection - gencon policy

As many of you may already know, garbage collection policies affect system stability and throughput. From Java5 and above, "generational garbage collection" policy was introduced with the motivation that most objects are temporary and die young, and need to be collected more frequently than long lived objects. However, simply setting JVM gcpolicy argument to gencon may not be sufficient. Recently I was investigating policy settings for some apps where the heap usage graph was a nice ramp, growing up to 900MB and scaling down to 300 MB; no issues related to performance, stability, etc., but the graph was little concerning. The clip off/shrinkage always kicked at around 900MB (red plot) on a 1GB heap max; what if a large request was received at that time; will the system crash or is this an expected gencon behavior. All these intriguing questions made me look deep into this generational collection policy.

        Gencon policy, splits your VM in two regions, 

- Nursery (aka young region) where new and ALL objects are created (except when new object cannot find an allocation in this region)

- Nursery is again split into 2 spaces, allocate space and survivor space. The allocation space being larger than survivor space.

- Tenure (aka old generation) where little older objects that are still referenced by your application and have survived from nursery, gets moved to.

           Garbage collection is more frequent in young region and are of short duration. Most objects do not survive and if they do, they get moved into survivor space and subsequently into tenure space after they survived multiple collections. So far so good, but then the following questions come across. How large should nursery be? What should be the ratio between allocation, and survivor space? Will the tenure size be fixed or vary? What will be the GC overhead? Without thinking through these, setting gcpolicy to gencon may produce undesirable effect, especially if your application attracts high volume.

         Back to our chart above, it is clear the gc policy was set without taking other factors into consideration. For a 1GB heap, by default young generation is sized at 64M and old generation to 960M and as a result large objects that cannot get into young region were allocated to old generation and this region has less frequent garbage collection explaining the slow ramp growth upto 900+MB. But this still does not answer the clip off exactly at 950MB each time. Looking into further, let us check heap expansion and shrinkage, a normal JVM occurrence, except that under this gc policy they need to meet some conditions. To shrink heap space, occupancy should be under 40% and heap has not been recently expanded and the object exists in the area being shrunk. The only explanation that fits the above chart is our usage level is less than 40% of total heap (evident from lower trough at 380MB).

       To conclude, gencon could handle a higher "rate of garbage collection", provide a shorter longest garbage collection cycle, and less frequent long garbage collection cycle. On the downside, It also means a net increase in memory usage due to multiple regions and if your GC overhead is > 10%, you have likely chosen the wrong one.

JVM Crash Analysis

It often is blamed on technology for application's poor performance, instability, poor scalability or even in some cases lack of usability, not realizing the blame lies within the application and its design; in other words improper use of technology. One such frequently encountered outcome from all poorly coded or designed application is JVM crash. There are many reasons for system to crash, the first step is to identify those before prescribing options or tuning the system. I have seen some developers and managers jumping into conclusion recommending software and hardware upgrades as a quick remedy. It is just a steroid before the effect fades with increase in usage or subsequent feature release. Performance engineering is an art and developers need to have an end goal even before analyzing and tuning, or else this exercise may go on for ever. For example, if you are recycling JVM every night, your goal could be to recycle every weekend. (in this case, you need to redesign your app for 24x7 availability, you simply cannot tune and get away from overnight recycle to no recycle). If your response time is slow, your goal will be to define and quantify slow vs speed and decide your options.

Thankfully for Java based platforms to perform diagnosis, core dumps provide you all necessary information to the level of active thread and its internal state at the time of the crash. If crash is frequent one may need to collect various type of dumps for analysis, such as heap dump, sniff dump, gc logs, and core dump. The first item to look for in core file is 1TISIGINFO, which provides clue to the type of signal that caused the crash. There are over two dozen signals in AIX and Unix world, and can be broadly grouped into four categories, exceptions, errors, interrupts and controls. Core dump from out of memory condition will normally raise SIGSEGV signal identifying the thread that caused the failure. This signal merely implies, an incorrect access to memory was performed by the thread OR in other words the thread tried to write into an inaccessible memory. It is not necessary for the heap space to be depleted, a common conclusion all newbies come into. Also, looking into 1XHCURRENTTHD one can track thread id and look into its corresponding monitors and heap locks that were active at the time of failure. Another useful, code will be 1STHEAPFREE, which provides free heap (in bytes) at time of failure; co-relating with requested heap size provides one with two choices, 1. heap starvation due to fragmentation, or 2. starvation due to deadlock.

Once identifying a choice, the next step will be to collect thread dump or gc logs depending on whether your deep analysis is to detect deadlock or heap fragmentation. Deadlocks are result of bad code logic or un-controlled wait times involved while interacting with external interfaces or systems. A classic example, I uncovered recently with my client was related to axis timeout. Our client application as a portal integrates with multiple reporting engines, and one use case involves sending instructions to purge inactive documents with encyclopedia (an actuate repository) folders via SOAP message, and based on outcome proceed and complete local transaction. All worked well until there was an increase in number of purge file records, when the request took longer to complete; the axis thread would listen to the socket and read bufferstream and in the process effectively acquired heap lock. Other threads working on various use cases while trying to write into heap, attempts to acquire heap lock, fails and enters into wait state. A normal scenario that a thread would enter and exit within milliseconds, but in this case the axis default timeout, around 1 minute, was a way too long for heap management which now can neither run a GC nor heapAlloc, thus resulting in deadlock. The outcome of course in this case was still an out of memory, because from a GC standpoint it cannot reclaim any memory.

In this above example, the crash was a result of bad program logic. In other situations you may notice sufficient heap space but still allocation failure (AF) would occur for object requests of few KB in size. This clearly is due to large heap fragmentation; taking an inventory of pinned and dozed objects in your heap will assist in tuning proper cluster regions within heap so that these unmovable objects get pushed to lower region of heap space. Note, Fragmentation cannot be eliminated but can only be minimized and apart from tuning JVM parameters, installing current service packs of JVM really helps.

Slower website under Akamai

This week I was investigating an interesting problem related to slowness of website for a particular client accessing from North Atlantic island, Bermuda. This client experienced slowness only while accessing our website in spite of having a 10Mbps pipe. To give a background, our website hosted in US runs on Akamai relay and has an average response time under 4 secs for most of the pages from across the globe. But that was not enough to convince our client who compared our site with competitors. Clearly the problem was on the client end, but how to convince?

To start, trace route was requested from client location to both websites; client trying to reach our website hopped across ISP's in europe, west coast, mid-west before reaching into ours in north east. This should not have happened since we were on Akamai. Next, looking at clients ISP provider our suspicion grew further; they were using a DNS provider in US, which computed a path closer to its location, i.e US. In a general web relay this is normal, but with Akamai this does not play well.

Content distribution networks like Akamai, InterNap, etc., deploy servers globally that contains copies of data placed at various nodes and communicate with their other nodes using high speed network and proprietary protocols. This speeds up page publishing for clients connecting from global locations, who do not have to connect to the website hosted in your data center, but rather connect to one of these nodes nearer to their location. In our case, client should have hit an Akamai server within their region, which in turn should stream contents at blazing speed. This did not happen, for one of the two reasons:

1. Name server used by client's ISP network either did not contain proper akamai location in their region OR

2. The DNS was living on a different region, which picked an Akamai edge server near its location instead of clients location.

Analyzing, the name server IP it was evident the ISP provider was using services from US, which when used to query our domain name resolved to an Akamai node in US; and the client was now traveling on low speed public internet across continent to get into Akamai server node, loosing all benefits of high speed network.

The fix here is not with Akamai, but with the client and their external DNS provider. Such similar issues may also arise with corporate networks spanning geographies, where DNS translation may be delegated within internal network to a central node which may or may not be near to the request origination location.

HTTP1.1 Browser Connections

Some interesting thing I learned this week when investigating web sockets was related to browser-web server connection pool. Applications developed in the past, including web2.0, while trying to speed up user experience, introduced techniques such as AJAX, and iFrames. However, web resources downloaded to browser was always limited to number of connections that could be established to A server domain; this limitation created by design for supporting dial-up users, varied between browsers. In IE 5-7, number of connections was always limited to 2.
    So regardless of how efficiently your AJAX calls are initiated or iFrames are stacked, your browser wait time is not going to improve a lot. A couple of ways to speed things up,

1. Use multiple domains for resources. i.e split you static resources across more than one domain and have all dynamic content served from one domain name.
2. Migrate to IE8, or Firefox 3, however for client facing internet apps you do not know your client systems.
3. For the sake of fun, modify your registry to allow concurrent connections by following microsoft instructions (http://support.microsoft.com/kb/282402). Beware, pushing the max connections beyond 10 may be interpreted as DoS attack by web server or F5 on the receiving end you may end up blocking your IP.

  

Browser	HTTP/1.1	HTTP/1.0
IE 6,7	2	4
IE 8	6	6
Firefox 2	2	8
Firefox 3	6	6
Safari 3,4	4	4
Chrome 1,2	6	?
Chrome 3	4	4
Chrome 4+	6	?

IFrame Techniques

Most of the web developers at some point or other would have used iFrames for inline presentation or to simulate parallel requests to server. Though handy these iFrames have some drawbacks and do not always behave the way we intend. This is mainly a scripting issue and how iFrame component is utilized within an application. First the drawbacks,

• Iframes block onload of the main page. i.e onload Event on the main page can only fire after all elements on the main page is loaded, including the iFrame and its contents. If iFrame content takes longer to load, then main page' onload event may sometimes never get fired.
• The main page and iframe share the same connection pool to establish connection with web server. 

The first issue is under developer control, whereas the later depends on the browser and may need changes to deployment architecture.

Normal Iframe

You all know this one. It's the default way to load an iframe and works in all browsers:

<iframe src="/path/to/file" frameborder="0" width="728" height="90" scrolling="auto">
</iframe>

Using the Normal Iframe technique will result in the following behaviour in all browsers:

• Iframe starts loading before onload of main page
• Onload of the iframe fires after all iframe content is done loading
• Onload of main page fires *after* the iframe's onload fires: the iframe blocks the main page onload !
• While the iframe is loading, one or more browser busy indicators show something is loading  

This technique is OK if iFrame does not take longer time to load.

iframe After Onload

Imagine you want to load something in an iframe, but it's not really important for the page. Or the content of the iframe is not immediately visible to the user because it's way down below the fold or hidden behind a link/tab. Consider deferring the loading of the iframe until after the main page is done.

<script>

//doesn't block the load event
function createIframe(){
  var i = document.createElement("iframe");
  i.src = "path/to/file";
  i.scrolling = "auto";
  i.frameborder = "0";
  i.width = "200px";
  i.height = "100px";
  document.getElementById("div-that-holds-the-iframe").appendChild(i);
};
 
// Check for browser support of event handling capability
if (window.addEventListener)
window.addEventListener("load", createIframe, false);
else if (window.attachEvent)
window.attachEvent("onload", createIframe);
else window.onload = createIframe;

</script>

The Iframe After Onload technique will consistently show the following behaviour in all browsers:

• Iframe starts loading after onload of main page
• Onload of main page fires without interference of the iframe: the iframe does *not* block the main page onload !
• While the iframe is loading, one or more browser busy indicators show something is loading 

This technique though does not block the main page has two main drawbacks
1. Site visitors will still see browser busy indicator(s) while the iframe is loading and that means that they will see those indicators for a longer period of time (versus Normal Iframe).
2.There is a good chance the user leaves the page before the iframe has finished loading.
 

Iframe setTimeout()

The objective is to load the iframe without blocking onload. However, this technique will often not have the desired effect in IE8.

<iframe id="iframe1" src="" width="200" height="100" border="2"></iframe>

<script>

function setIframeSrc() {
  var s = "path/to/file";
  var iframe1 = document.getElementById('iframe1');
  if ( -1 == navigator.userAgent.indexOf("MSIE") ) {
    iframe1.src = s;
  }
  else {
    iframe1.location = s;
  }
}
setTimeout(setIframeSrc, 5);
</script>

In all browsers but IE8, the Iframe setTimeout() technique will consistently show the following behaviour:

• Iframe starts loading before onload of main page
• Onload of the iframe fires after all iframe content done loading
• Onload of main page fires soon: the iframe does *not* delay the main page onload
• While the iframe is loading, one or more browser busy indicators show something is loading 

This last option seem to provide both parallel loading and non-blocking of main page. However, with every browser version bringing in new features one should not be surprised if the sequence listed above varies for any particular browser version.