JVM Crash Analysis

It often is blamed on technology for application's poor performance, instability, poor scalability or even in some cases lack of usability, not realizing the blame lies within the application and its design; in other words improper use of technology. One such frequently encountered outcome from all poorly coded or designed application is JVM crash. There are many reasons for system to crash, the first step is to identify those before prescribing options or tuning the system. I have seen some developers and managers jumping into conclusion recommending software and hardware upgrades as a quick remedy. It is just a steroid before the effect fades with increase in usage or subsequent feature release. Performance engineering is an art and developers need to have an end goal even before analyzing and tuning, or else this exercise may go on for ever. For example, if you are recycling JVM every night, your goal could be to recycle every weekend. (in this case, you need to redesign your app for 24x7 availability, you simply cannot tune and get away from overnight recycle to no recycle). If your response time is slow, your goal will be to define and quantify slow vs speed and decide your options.

Thankfully for Java based platforms to perform diagnosis, core dumps provide you all necessary information to the level of active thread and its internal state at the time of the crash. If crash is frequent one may need to collect various type of dumps for analysis, such as heap dump, sniff dump, gc logs, and core dump. The first item to look for in core file is 1TISIGINFO, which provides clue to the type of signal that caused the crash. There are over two dozen signals in AIX and Unix world, and can be broadly grouped into four categories, exceptions, errors, interrupts and controls. Core dump from out of memory condition will normally raise SIGSEGV signal identifying the thread that caused the failure. This signal merely implies, an incorrect access to memory was performed by the thread OR in other words the thread tried to write into an inaccessible memory. It is not necessary for the heap space to be depleted, a common conclusion all newbies come into. Also, looking into 1XHCURRENTTHD one can track thread id and look into its corresponding monitors and heap locks that were active at the time of failure. Another useful, code will be 1STHEAPFREE, which provides free heap (in bytes) at time of failure; co-relating with requested heap size provides one with two choices, 1. heap starvation due to fragmentation, or 2. starvation due to deadlock.

Once identifying a choice, the next step will be to collect thread dump or gc logs depending on whether your deep analysis is to detect deadlock or heap fragmentation. Deadlocks are result of bad code logic or un-controlled wait times involved while interacting with external interfaces or systems. A classic example, I uncovered recently with my client was related to axis timeout. Our client application as a portal integrates with multiple reporting engines, and one use case involves sending instructions to purge inactive documents with encyclopedia (an actuate repository) folders via SOAP message, and based on outcome proceed and complete local transaction. All worked well until there was an increase in number of purge file records, when the request took longer to complete; the axis thread would listen to the socket and read bufferstream and in the process effectively acquired heap lock. Other threads working on various use cases while trying to write into heap, attempts to acquire heap lock, fails and enters into wait state. A normal scenario that a thread would enter and exit within milliseconds, but in this case the axis default timeout, around 1 minute, was a way too long for heap management which now can neither run a GC nor heapAlloc, thus resulting in deadlock. The outcome of course in this case was still an out of memory, because from a GC standpoint it cannot reclaim any memory.

In this above example, the crash was a result of bad program logic. In other situations you may notice sufficient heap space but still allocation failure (AF) would occur for object requests of few KB in size. This clearly is due to large heap fragmentation; taking an inventory of pinned and dozed objects in your heap will assist in tuning proper cluster regions within heap so that these unmovable objects get pushed to lower region of heap space. Note, Fragmentation cannot be eliminated but can only be minimized and apart from tuning JVM parameters, installing current service packs of JVM really helps.