Intermittent HTTP 500 Error (Response Code)

In this blog post, I would like to write about intermittent HTTP 500 Error happening in AJAX based application. This error is not the typical internal server error thrown by the application server, which also shows as HTTP response with 500 as response code. Typical symptoms are that the request did not even reach the back-end server, bu the 500 response code is returned.

The application is an AJAX application, built using GWT in the front end and Spring in the back-end. The application was accessed using Internet Explorer in the client desktop.
The back-end infrastructure has Apache Web Server, routing the request to the Websphere application server. The application is a heavy user interactive application which make makes multiple RPC (Remote Procedure Calls) to the back-end to fetch the data . This essentially means that the application was making large number of HTTP requests to communicate with the back-end server.

The issue that we were facing was that 1% of the requests was getting a 500 response code.

We tried analyzing the reason for the 500 response code.To understand the issue, we need to know the details about the communication between the browser and the server. When the browser makes an HTTP request to the server, the browser sends the HTTP Request Header and the HTTP Request Body. The Web-server at the back-end, receives the header and body, transfers it to the back end server. In the case where the HTTP 500 status code was returned to the application, it turned out that the error was thrown by the Apache Web-server. Apache has a thread (connection) waiting for the request header and body to arrive and then forward it to the back-end server. In very rare scenarios, the header of the request arrives and the request body never arrives. Apache waits for 1 minute (timeout duration) and then responds with a 500 Error Code.

The next challenge was to find out as to why the request body was not arriving and only the request header was arriving in some specific cases.
We found that this was happening only with the Internet Explorer browser (IE6/ IE7 & IE8) and not with Firefox. After placing a network monitoring tool at the client side, we figured out that the issue was happening with IE when it was trying to retransmit the data on a new connection after the attempt to send data on the first connection fails. In the scenarios we saw, IE was trying to send request to the back-end server on an open connection (HTTP 1.1 compliant) with the back-end server. However this connection was already closed by the back-end server , after the keepalivetimeout period was reached. IE, without knowing that the connection is closed, attempted to send the data on that connection. Per design, IE should then retransmit the data on a new connection with the back-end server. In some scenarios, IE forgets to send the HTTP Body and sends only the HTTP Header. (IE Bug #). Though this IE Bug is reported only in IE6, we found that this can occur in all versions of IE including IE8.

There are multiple fixes to the problem

The first and the preferred one is the one recommended by Microsoft in the bug details. This involves registry changes and a headache of applying it across the end users machines.

The second fix is to ensure that the IE is made to work with only valid connections (i.e connections not closed by the back-end server). This is possible by increasing the keepalivetimeout value in the back-end to a value more than 60 seconds. The 60 seconds value is due to the fact that IE has a default setting of 60 seconds after which it removes the unused open connections with the back-end server. However, a higher value of KeepAliveTimeout also means that there would be large number of idle connections in Apache which would be reserved to a client.

Oracle Truncate Table - Storage Clause Significance

This week I performed a truncate on a table in Oracle database to recover some space in my tablespace. Even after truncate the space was not released back to the tablespace. There I realized the storage clause of the table as well as the truncate command is important.

The storage clause in the truncate command is required to be specified while performing a truncate command. The storage clause determine as to what will happened to the space released by the rows that got removed. When a truncate is issued on a table, Oracle Deallocates all space used by the removed rows except that specified by the MINEXTENTS storage parameter (be aware that the de-allocation depends on the min extents clause and be wary when the table is large and is imported)

The options are

Drop Storage – This means that all space used by the object would be released except for the space allocated by the minextent parameter. This space can be used by any other object or by the same object from which the data is truncated. However, if the minextent (along with the initial value) is large enough, this space is NOT released even after the truncate. For e.g. See the storage parameter for a table that got imported into my database

STORAGE (
INITIAL 2432M
MINEXTENTS 1
MAXEXTENTS UNLIMITED
PCTINCREASE 0
BUFFER_POOL DEFAULT
)

If the table with the above definition is truncated, 2432M is not released back to the tablespace for other objects to use. However, if you perform alter table move tablespace X or Drop table, the space would be released back to the tablespace.

Reuse Storage – This means that the space is not released back to the pool instead the space is reserved for the same table. Hence if the table is occupying 50M, a truncate would not release the 50M back to the tablespace instead the space would be reserved for the same table. This option needs to be used in case you are sure that you are going to reload the similar data again into the same table.

High CPU usage of database

In my experience ,Oracle database is capable of taking good amount of load and can utilize the hardware power very effectively for scaling. However in some case , you might find that the CPU usage by the database is considerably high. In my experience , these factors can also cause a high CPU utilization

• Queries performing excessive work to achieve desired results. This could be more data blocks scanning, more nested loops etc.
• Table setups can be cause of concern if there is large amount of row chaining etc.
• Large amount of house keeping work on database like archival sweep, rman processes etc.
• Application keeps opening new physical connections to the database instead of using the connection pool. Even though application servers configure the connection pool, the connection pool might open new physical connections for all transactions due to wrong set up of connection pool.
• XA Transaction has a higher impact on the database as it needs to perform the house keeping activity like managing the transactions stages like prepare, commit etc .
• When less number of CPUs are present and more number of processes are present on the system, the system usage will go high due to context switching.
• As a matter of fact, the CPU utilization will go invalid or the system starts behaving in an awkward manner when the CPU utilization hits above 85%.

In some cases, correcting the above symptoms would fix the high CPU usage. However , in many other cases, it might be that the applications needs that much processing power to support the database operations.

Article on Cloud Computing

My first article on the web got published few weeks back. The article can be found @

http://ajax.sys-con.com/node/938581

The article concentrates on the online software distribution using the cloud.

Hope you enjoy reading it.

Multithreading - Downstream Impacts

Multi threaded model will in most of the cases give better performance for batch applications. I would look at the below mentioned points along with the reengineering of the application code.

• Processing Power – There should be enough CPU power available in the machine.
• Memory – Since all threads will be working in parallel, the amount of memory used also will be considerably high. This can be however be reduced by not placing too many objects in JVM. However, the amount of memory required will be proportional to the number of threads in the application.
• Network Load – If the application is having network interactions like Database queries, FTP etc, the network load also will be high. In some cases, it would be solved by adding GBit connections between servers communicating. In most of the cases, the normal NIC’s itself can process the load.
• Disk Speed – If the application has lot of File processing, the disk accesses also need to be tuned. It would be better to read the files from SAN rather than NAS as the disk response time is better on SAN (my experience).
• Database Setup – If there is lot of database interactions, the database also should be made aware of such a change in the application. The load on the database will increase as there will be multiple threads that will try to fetch data from database. Most probably, it will end up increasing the database parameters to accept more loads.
• GC processing – The GC tuning should be performed. Since many threads are working in parallel, the amount of garbage created also will be high. An effective tune up of GC is required, without which, application will end up in OutofMemory Error. You can consider Parallel GC but ensure that the number of GC threads is mentioned.
• Synchronization – Objects that are created in JVM scope should be accessed with proper synchronization. If this is not worked out properly, it can cause dreaded issues like data corruption, deadlocks etc…

Multithreading Multiprocessor Relation

A batch application is made scalable by ensuring that the executable can use the complete power provided by the machine. The batch application should be designed as multithreaded model if it’s possible to break the work into multiple smaller units of work. In this way, each thread can work on its own piece of work and complete the work. For e.g. In a single threaded model, the batch processes takes 10 hours to process 2000 customer records. If the same code is written in multi threaded model using 10 threads, the job can be split into 10 units with each unit having to process 200 customers. The same work can be completed in 1 hour. Caveat being the machine has the necessary processing power (CPU)

Any java process executes in a thread of execution. The thread can perform multiple activities like performing the task, waiting on IO, waiting on socket, waiting for lock release etc. While the thread is waiting on something, CPU is intelligent enough to remove the thread from its cycle and take up another thread which can perform the work. Any point in time, a CPU core can execute only one thread. So if the machine has 4 cores of CPU, an ideal count would be to provide 3 threads/core for the application to use. The number of threads per core is dependent on the application, primary driving factor being what is done in each thread. If there considerable wait that will happen in the thread of execution (like File IO, database read, Socket read) , the number of threads per CPU can be increased and if the thread is going to perform operations within the process area without any wait, the number of threads per CPU should be reduced. This is because the machine always pushes out the threads that are waiting and takes in thread that is ready for execution.

Impact on number of threads

More threads/core for application that has less amount of wait.
Let’s take an example where the machine has 2 CPU’s and application is configured to use 10 threads. Since there is not much wait time involved, the CPU will force the executing thread out of its cycle to give fair chance for the remaining 9 threads to execute. The thread that got pushed out will come back to execution after certain CPU cycles. At this time, it needs to rebuild till the point where it was pushed out. If there were only 1 thread of execution per CPU, this type of activity won’t happen and the single thread /CPU can complete the operation without heavy context switching. In such a scenario, it will be detrimental for the application. Such a case will be evident if the batch completes in lesser time when the number of threads for the application is reduced.
Less threads/core for application that has considerable amount of wait.
Let’s take an example where the machine has 8 CPU’s and application is configured to use 8 threads. Each CPU will execute a thread of execution and when any one of the thread goes into WAIT state, the CPU lies idle. Such a case will be evident if the batch completes in lesser time when the number of threads for the application is increased. At any point in time, the CPU usage will not be near 50% or 60%.

As mentioned in the above description, the number of threads per CPU should be decided based on the application characteristics.

Avoding log4j logger statements in log files

We all have used log4j in multiple scenarios and it serves the purpose. There are cases when we don’t want the logger statements from few classes or few packages to clutter the log files. Some other cases, we want the logger statement to be printed in only one log file and not in all log files. These are 2 different cases and I will explain solution for both of the requirements.

Requirement 1: Don’t print the log statements in any of the configured log files. Consider a scenario where we have configured 4 -5 log files at different levels and for different purposes. If we don’t want the logger statement from the classes in a particular package or particular class to appear in any of these log files, we have to use the following syntax.

log4j.category.=OFF

e.g. log4j.category.com.test.welcome=OFF means that all the statements logged from the classes present in com.test.welcome package will not be logged in any of the configured log files.

Requirement 2: Don’t print the same log statements in all of the configured log files. It should be printed in the more specific log file only. Consider the scenario like this

log4j.category.com.pack1.pack2 = INFO, appender1

log4j.category.com.pack1 = INFO, appender2

The statements logged from the classes present in the package com.pack1 will be printed in the appender2 as well as appender1 log files as both of them satisfy the category. Now if we don’t want the logger statements from the pack2 not to print in appender1, then we can use the additivity clause.

# set additivity to false to ensure that the parent appenders do not log these statements
log4j.additivity.pack2 =false

The above line means that the logger statement from pack2 will be present only in appender1 log file and not in appender2 log file. This is because the logger file is configured for appender2 as well as appender1. That means the logger statement got printed only in the specific log file and not in its parent appenders.

RAID

Data is the most important part of any organization. So the organization takes care in preserving the data by all means. The daily data is backed up. The data is also backed up into tapes and the tapes are stored in a distant location. In case of any data center disaster, the data is recovered from the alternate location. These solutions works in recovering the data till the last backup was performed. Many verticals like banks would not be happy with this approach.

Redundant Array of Independent Disks is a way to preserve the data to the latest point. It is sometimes referred to as Redundant Array of Inexpensive Disks. RAID deals with the configuration of the data storage disks. The storage disks will be in smaller blocks and the data will be written into these smaller disks, controlled by the RAID controller. For e.g. if a storage of 500GB is required, it can be obtained by stripping (combining) 7 units of 72GB disks. The controller will have the responsibility of allocating the data into these 72GB disks. For the end user, in this case an OS, it appears like a single 500GB disk. The advantage is that the writing to the disks will happen in parallel and in this case will achieve 7 times faster write speed than a single 500GB disk.

RAID0 – RAID 0 means stripping to get a bigger disk. There is no backup or crash recovery mechanism present in this type of setup. Many smaller disks will be stripped to make a bigger disk, controlled by a controller. This set up cannot sustain a disk failure.

RAID1 – RAID 1 means mirroring. The disk is mirrored. The redundancy of the data is present but in case of a crash of any of the disk, the data can be recovered from the mirrored disk. With higher redundancy and poor performance, data safety from crash is achieved. Failover is also achieved in this setup.

RAID 3 – RAID 3 is a case where the data is stored in one location and its parity is stored in a different disk. This works with 3 disks where the data will be written to disk1 and disk2 and the odd/even parity will be written to the parity disk. In case the disk1 fails, using the parity information and the data on disk2, the data on disk1 can be recovered. In this setup, the data redundancy is 33%. The issue in such a setup was that the parity disk had a huge load to handle.

RAID5 – Similar to RAID3 but in this case, the parity disk is rotated between the data disks. In RAID3, we allocated the 3^rd disk as the parity disk. In this setup, the parity disk is rotated between all the disks. For the first time, the 3^rd disk will have the parity information and 1^st and 2^nd will have the data. For the next data written, the 2^nd and 3^rd will have the data and the parity will be stored in the 1^st disk. This continues. A single disk failure can be sustained with this set up without having 50% redundancy like RAID1 but still having 33% redundancy like RAID3.

RAID6 – RAID6 works on dual parity. In this setup the parity information is stored in two disks and the data is also stored in 2 disks. The parity disks are rotated between the available disks like RAID5. RAID6 set up can sustain 2 disk failures. In case 2 disks fail at the same time, the data in those disks can be recovered using the parity information stored in the other 2 disks.

RAID0+1 – RAID 0+1 is merging RAID 0 and RAID 1 in the same order to get the advantages. Firstly all of the available disks are set up as RAID 0 i.e. all the disks are stripped together to achieve the required space. Another exact configuration is done with another set of disks. These to configurations are then configured as RAID1 i.e. mirrored. In other words, it’s setting up 2 RAID0 set up and then performing RAID1 setup on it i.e. mirroring. This set up can sustain the disk failure of any disks as long as the mirror copy is intact. However there is a 50% redundancy of space.

RAID1+0 – RAID1 + o is merging RAID1 and RAID0. The individual disks available are first mirrored. Then these mirrored disks are then stripped together to achieve the required space. This setup can sustain any disk failure (as long as the mirror doesn’t fail) but has 50% redundancy.

Based on the space redundancy, cost implications can be figured out. Based on the possibility of recovery, the availability can be measured. Similarly performance can be measured based on how many places the data needs to be written. RAID1 is better performing as data is written to a single disk and no mirror copy is written but of course with lesser availability. Similar analysis can be done for other RAID levels too.

What is SAN & NAS

When an enterprise class machine is procured from a vendor like HP, IBM or DELL, it doesn’t come with a hard disk to store your data. The hard disk that is shipped with the machine is primarily used for loading the operating system. The capacity of such a hard disk will be generally 72GB, which will be mirrored if procured that way. So where can the users store their data? It can be stored in two types of places, the SAN and the NAS.

Storage Area Network means the storage is a separate entity that is connected to the machine through a separate network. This network will be different from the LAN. The storage area will be connected to the machine using a Fiber channel which gives a considerable speed in transferring the data. SAN uses block storage access which gives better performance. Also the data stored in the SAN area is accessed by disk numbers and it’s the raw data. This type of storage can be used for database storage. The SAN allocated to one machine cannot be accessed by any other machine and is not suitable for sharing data between machines.

Network Attached Storage is where the storage is in a different location and data transfer happen through the LAN. The impact on the LAN will be high when the data is transferred to the NAS. The data placed on the NAS is accessed by filenames and byte offsets. The security and permission on the files are also handled by NAS itself. For this reason, the accessing of he files will be slower than SAN. NAS is ideal for sharing the data between two machines even if the machine is using different operating systems. This type of storage can be used for file share servers, database backups, emails storages etc ….

Capacity Planning - Series 1

Capacity planning is the exercise of determining the capacity required for the organization to meet the needs of the software application to in a cost effective manner so that the application performs within accepted and agreed Service Level Agreements. This science is not a precise science when done.

Im sharing my experiences in capacity planning through this series. When I started working on Capacity planning and took few books to read it, it was filled with lot of mathematical formulas to arrive at the numbers. So a warning note is that if you are interested in Capacity planning, you should be ready to deal with equations (yes, it can be solving linear equations, solving etc, which we have studied during our engineering courses).

There are two approaches of performing capacity planning.

Case i: If the application is not yet developed and implemented

Case ii: If the application is developed and we have a testing setup where the application can be subjected to test load.

In my working experience I have not done much of the case i and hence I would not be dealing with that section in detail. However you should be comfortable with the Utilization law and the queuing models to arrive at a good estimation.

This article will talk about how to proceed with capacity planning with situations that arise of Case ii.

Capacity Planning exercise will have to perform the following activities which will be required as Input to the calculation.

1. Workload Profile – The workload can be defined in terms of transactions for online processes and as processes for Batch Processing. For an online application, the workload can have the following information
1. Details about the user groups
2. Details about Peak load timing
3. Details about the various business transactions as well as its timing
4. Details about the non business requirements like backup, archiving etc..
5. Details about user locations.
2. Expected SLA – The expected SLA should be captured. You should always try to get quantitative figures instead of qualitative figures like “good”, “perfect” etc. The SLA should also be defined for Peak load timing too.
3. Future Growth – The growth pattern for the application as well as its usage should be collected. This will ensure that the application will function well in future and still meet the expected SLA.

Capacity planning exercise should have the following details as its output.

1. Servers – Brand, Numbers, Location, Distribution and Load Balancing details.
2. Disks – Stripping, Speed of the Disk, Capacity of Disks , Partitioning of Disks, Type of Disk (Clarion/Symmetrix)
3. CPU – Type (AMD/Intel etc), Speed, Cores, Memory Cache, Number.
4. Memory – Type, Amount, Expansion Slots, Max Memory.
5. Networks – Expansion Slots, Channels, Network Cards (Embedded NIC’s)