Stress testing a web/application server is never an easy task. Having never done any intensive server testing before I got into load testing our e-library server with some obvious but important questions in my mind. What kind of tools are available? What kind of tests can i perform? Will I be able to make sense of the data from the result? and Will it actually apply to real scenario–will my server behave exactly the way it would during the test?
These are simple questions for an experienced administrator, but for someone just getting into performing such test, it seemed a fairly daunting task. So lets start with the things we want to test and the server side applications to test.
The e-library is an online library system that run on Ubuntu 7.10 desktop (I am planning to change this to Ubuntu 8.04 server–mainly because I don’t want the desktop applications to take up valuable
memory away from the server applications). The e-library uses fedora commons as its library engine, and has Fez as its GUI. Fedora commons also uses Tomcat a Java application that talks to apache, but can also act a server on its own. More on “e-pustakalaya” or e-library later.
So the things that we really want to test are.
1. Avaliablity of the web-server: One of the main thing (among others) that we want to make sure that apache does or does not– depending on the way you look at it– is the way it handles max client and the keeepalive time for clients connected. It is a good practice to set the max clients to a high number ( = total memeory/5Mb), and keep alive time to as low as possible(=2). Take a look at this virtualthread blogpost for a good overview of how these parameters effects your server.
2. Resource Utilization: One thing i have noticed is that my server had a huge overhead on CPU and my memory was down to 12Mb free, which caused the server to crash. This happened when i simulated a 100 user connection making one new connection every 10 sec….fetching a 1.4 Mb pdf file. So hardware really matters, and it is really a good idea to optimize apache and any application service for your hardware specs. Reading memory usage and CPU load is a tricky thing. I found out that the memory information that say $top gives for a process is not actually correct view of what that process is doing. Similarly, making sense of load average and CPU% requires a little more knowledge of the Linux system.
Load average = Is the CPU load. It gives information on the number of processes waiting to run + the number of processes executing. It is not about CPU utilization but the total queue length.
This article from Linux journal gives a nice idea about load average.
3.Response time test, Throughput test and Error rate test: These are probably the main areas to focus while performing a load test. To do these one should look into one or all of the following.
a)User-based testing: Where you get many users to use your server and monitor the performance.
b)Standard benchmarking: I used Apache benchmarking tool for this. Apache benchmark can give you an idea of how many request per second your apache server can handle.
$ ab -n 10 -c 10 http://www.yourdomain.com/index.html
Will have 10 concurrent users and 10 request of get index.html on your web server.
$ab -t 60 -c 10 http://www.yourdomain.com/index.html
Will have 10 concurrent users for 60 secs with as many hits as possible, by default when you
don’t use the -n option, ab sets to default 50,000.
You can also use a -g option to get the output on a txt file to use for gnuplot.
$ab -n 10 -c 10 -g test.txt http://www.yourdomain.com/index.html
From Apache website:
“Apache JMeter may be used to test performance both on static and dynamic resources (files, Servlets, Perl scripts, Java Objects, Data Bases and Queries, FTP Servers and more). It can be used to simulate a heavy load on a server, network or object to test its strength or to analyze overall performance under different load types. You can use it to make a graphical analysis of performance or to test your server/script/object behavior under heavy concurrent load.”
So lets take a look at some of the thing we can do with JMeter. After the installation of JMeter, run it under a Linux system with /path to your jmeter/bin/ ./jmeter
A JMeter window will open, which looks something like this:
Use the Options->Expand All option to see your full tree.
Now set up a test scenario (refer to JMeter user page). After you run the test you will see (if you use the output graph result as one of the listener) something like this (Two pictures for two different run with same number of users and loop).
So when I am doing stress test, I am mostly paying attention to the throughput and average output results for now. Needless to say you want your server to handle as many request per minute with less error rate (you can check this by adding a summary report to your listener)and high KB/sec throughput, but you also want to see consistency. If you look the two runs it shows a similar result. Thats exactly what we want to see. We want to see a consistent pattern for load, memory and other things. Any erratic behavior should indicate a poor performing server. I would say its a good idea to check both load with $top or $uptime and memory and cpu usage while performing these tests. This will give you an idea about your server’s threshold before it overloads. JMeter is pretty cool because you can run different kinds of test on your server. Once again check out the JMeter page for more information.
Other tools for performance and functional testing can be found at opensourcetesting.
For Load Balancing:
1. mod_backhand is used to redirect requests from one server to another. Is used for cluster load balancing.
2. Load balancing with cluster apache setup with Round Robin DNS.
Say you setup two machines with ip 192.168.1.1 and 192.168.1.2.
Now on the DNS server have two entries for your server:
www.yourserver.com A 192.168.1.1
www.yourserver.com A 192.168.1.2
The DNS will distribute incomming requests between these two.
Any corrections to these or new ideas and methods for server load/stress testing is welcome.