This article discusses how to build a portfolio of scenarios for load testing automation. You have rarely only a single scenario to test, so you have to decide which scenarios to include in your test plans. Criticality, frequency, difficulty and verifiability are the four key factors that should be considered to determine the scenarios to automate.
Author: Christopher Merrill, Web Performance
Originally published in September 2010 on Web Performance Blog
What to test?
Occasionally, we encounter customers who have only a single scenario to test. For example, one client developed an application in which the only scenario of interest involved a user registering him/herself with the system and scheduling an appointment. When this happens, the tester may devote their efforts to the accurate simulation, testing and analysis of this single scenario. You are unlikely to be so lucky.
In any moderately complex system, there are dozens or even hundreds of scenarios that are candidates for load testing. You are not likely to have the time or resources to test them all. As a result, you must make some tough decisions about which scenarios will be included in the test plans.
There are a number of key factors that will raise or lower the importance of each scenario:
* Criticality – How critical is the scenario to the overall operation of the application?
* Frequency – How often is the scenario expected to be performed?
* Difficulty – How difficult is the scenario to simulate?
* Verifiability – Can correct operation of the scenario be easily verified?
The first two factors, criticality and frequency, should be the keys in deciding IF the scenario is tested. Scenarios that are both critical and frequent must be load tested. Those that are neither will typically be left out of the testing process entirely. The remaining scenarios, those that are somewhat critical or somewhat frequent or a little of both should be tested as the schedule allows.
We use the second two, difficulty and verifiability, to determine WHEN the scenario will be tested. This is particularly true at the beginning of the testing effort when the tester is not yet experienced with the application, tools and/or testing process. The first scenarios to be tested should be those that are easy to simulate and easy to verify. Unfortunately, at this point the inexperienced tester is not likely to know which scenarios will be difficult to simulate. You will have to substitute complexity as a measure of simulation difficulty until that knowledge is gained. As an example, in a CRM system, a scenario that consists of logging into the system and sending an e-mail is both relatively simple and easy to verify.
You may be asking “Why does the ease of testing or verification matter? Shouldn’t we always be testing the most critical and frequent testcases first?” The answer is a resounding “No!” If we lived in a world with no schedules, I might be inclined to agree with you. But this is the real world. Besides the realities of scheduling, there are two other important facts to take into account. First, bugs cost less to fix earlier in the development/testing cycle. Second, a large percentage of performance problems are related to system configuration, framework limitations or other system-wide factors. In these cases, exercising nearly any scenario will uncover the problem.
To put that in plain english: testing 5 less-critical or less-frequent scenarios this week is better than testing 2 critical and frequent scenarios next month. If your organization does not have significant experience with load testing, this effect is amplified, since the testing schedule will be further extended as a result of the learning curve.
Our Preferred Procedure
Summarizing from above, we generally follow the procedure:
1. Pick one or two simple, easily verifiable scenarios and test them. We use this phase to ensure that the testing infrastructure is in place and operating properly.
2. Add scenarios that are easy to simulate and either frequent or critical to the test suite.
3. Add the most critical and most frequent scenarios, regardless of difficulty, to the test suite.
I hope this helps you plan your load testing more effectively.