It has been put in evidence by some famous recent failures that load testing is an important activity when you release a public web site. This article provides some hints and tips hat will allow you to prepare for successful load tests. These load testing best practices range from working with the right team to collecting the meaningful metrics.
Andrew Smith, BugHuntress, http://www.bughuntress.com/
Being an integral part of every development effort, load testing aims to define the limits and capacity of the application by putting demand on a system and measuring its response. The goal is to find out the maximum amount of load a system can handle without significant performance decrease.
To be performed successfully, load testing should be done in accordance with its basic stages. When starting load testing, you should keep in mind an approximate plan of steps to take. A tester should:
1) outline the objectives of the app;
2) figure out the key scenarios;
3) highlight the workload;
4) identify metrics;
5) choose a load testing tool;
6) make test cases;
7) learn the load environment for your testing;
8) run it gradually;
9) gather the results and make analysis.
The load test will only reach its goal if you rigorously prepare to it. The more precisely you model the real world conditions, the more accurate and reliable the test results will be.
Below are several tips to prepare for optimal load testing:
- Know how the test tool works. Read the documentation, practice, check out what all the settings do, and then figure out how users are using the application.
- Collect realistic usage data. Non-realistic data can result in the wrong outcome of the load test.
- Write a test plan. You may already know what you will do, but writing a test plan will save your time and keep your project from unexpected things.
- Monitor your test environment infrastructure and control the changes. Some testers don’t do it, but if they do, it will help to define the problem easily by monitoring the servers. Keep tracks of the changes being done to your environment.
- Estimate the number of users. The number of virtual users to simulate should be a shot in the dark. Ask marketers, architect and engineers how many concurrent users they designed the application for. Test this number and a little above it. Mind that this number may differ depending on the time of the day.
- Manage your team. Gather the right people for conducting the test: database administrator, developer, technical architect and others. They will approach the problem from different angles and provide you with a balanced feedback and solution.
- Configure the application and servers for testing. This includes switching on caching, settting an appropriate logging level, etc.
- Plan the time to analyze results and implement changes. As you have a team for load testing, notify everybody of analyzing test results as a group in advance. Also, you should schedule the time for implementing changes. It is important to understand that various remedies such as implementing a caching strategy, database optimization, code refactoring and others can require a wide range of time and costs to be implemented.
- Mind user behavior on your web site or app. The test scripts should reflect the activity of your users and types of transactions a web site is expected to handle. For instance, not every visitor coming to your site would buy something. This should be correctly represented in the scenarios for load test.
- Include think time data to your test scripts. Different users spend different period of time on some page, and the duration of this pause should be considered when recording or writing load testing scripts.
- Mind client platforms and preferences. Various browsers or versions have different defaults that might affect workload that a browser puts on a Web server. Moreover, a user can change his default settings himself. Although it is a rare occasion, a more accurate load test would vary these values.
- Take account of geographical locations. Firstly, the user location usually defines the browser he uses, thus you should prepare the most popular browsers used in a certain region or country. Secondly, Internet response time can vary from city to city depending on the country, on the distance between the client and the host web site, as well as on the time of day, local network capacities, etc.
Metrics
One of the important steps when preparing to a load test is defining metrics. This is the way to quantify the results of your tests. The most important load testing metrics are:
- Response time – the total time it takes after the client sends a request till it gets a response. Response time includes:
– page load time;
– time to receive the first byte from the server;
– time to receive the last byte from the server;
- Number of transactions passed/failed.
- Throughput – the amount of data in bytes that users receive from the server at any given second.
- Load size – the number of concurrent simulated users trying to access the application.
- CPU utilization while processing the request.
- Memory utilization while processing the request.
The Choice of a Load Testing Tool
There are many tools that can be used for load testing of software projects. They create a certain workload and measure system response under this load, providing as a result the metrics discussed above. The choice the tool depends on your needs, your budget and project size. In general, you have a choice between open source and commercial tools, as well as local tools or hosted tools that provides load testing as a service.
Conclusion
Every major application should pass load testing before going live. Being an extremely essential activity, it requires a well-planned approach and execution. Not only it allows to discover interesting insights on why a web application can fail, but also means continuous improvement of your product.
References
* Software Load Testing on Wikipedia http://en.wikipedia.org/wiki/Software_performance_testing
* Real-World Load Testing Tips to Avoid Bottlenecks When Your Web App Goes Live
http://msdn.microsoft.com/en-us/magazine/cc188783.aspx
* Load Testing in 10 Steps http://www.codeproject.com/Articles/94007/Load-Testing-in-Steps
* 9 Tips to Prepare Your App for Optimal Load Testing http://blog.smartbear.com/load-testing/9-tips-to-prepare-your-app-for-optimal-load-testing/
* Modeling the Real World for Load Testing Web Sites http://www.methodsandtools.com/archive/archive.php?id=38
* Testing Performance of Mobile Apps – Part 2: A Walk on the Wild Server Side http://www.methodsandtools.com/archive/mobileserverperformance.php
* Load Testing Tools Directory http://www.loadtestingtools.org/
About the Author
Andrew Smith is working for the BugHuntress software testing company which is specialized in testing web, mobile and desktop applications. Andrew enjoys writing about software testing, mobile and IT technologies.
Thanks for the article, very detailed guide to prepare for load testing. I’m interested in metrics, I mean – what range of values is taken to be high, middle and low?
All the metrics have very relative values and depend on many factors. 1. Response time: when many users log in to the site, the high value of response time (or page loading time) would be up to 3 seconds, low value – more than 4 seconds. 2. Load size depends on the amount of users, number of requests and the amount of requested data. The more these values are, the higher load size is. 3. Number of transactions passed/failed: ideally, all the transactions have to pass. 4. CPU utilization, Memory utilization: these metrics are monitored on the server and Load size directly affects them. Ideally, the server must have enough CPU and memory to cope with maximum load. 5. Throughput depends on throughput capacity between the server and the computer, and the responses should be optimized enough to deal with inquiries of thousands of users.