Software tests never run fast enough. To improve this performance, this article presents a process called DMAIC (Define, Measure, Analyze, Improve, Control). It shows how to apply this approach with Cucumber, an open source Behavior Driven Development tool. Cucumber lets you describe how software should behave in plain text. The text is written in a business-readable domain-specific language and serves as requirements documentation and source of automated acceptance tests.
Author: John Sextro, Agile Coach, Asynchrony Solutions, http://blog.asolutions.com
Let’s face it, you can never have enough money and your tests never run fast enough. I can’t help you with your money, but if you follow the steps in this article I can help you speed up your Cucumber tests.
The improvement process that I’ll walk you through is known as DMAIC (Define, Measure, Analyze, Improve, Control). DMAIC is an improvement cycle commonly used in Six Sigma to improve and optimize business processes. Let’s take a look at how it works.
Define
You’ve probably already defined the problem and have determined that your tests are too slow. What you haven’t done is define success. You will define success in terms of execution time. For instance, if your Cukes are taking 30 minutes you might define success as, “Make all Cukes run in 20 minutes or less.” This target is up to you and your team to decide. Just remember, the amount of time it takes to run your tests will be inversely proportional to the number of times you will run them. So the faster they are, the more often you will run them.
Measure
You know how long it takes to run the suite. You might even know how long it takes to run each scenario or sets of scenarios. What you need to know now is how long it takes each step to run. But wait. Before you go diving back into your steps to instrument them with timers, let’s step back and see what Cucumber can do to help us.
Fortunately, Cucumber offers a number of helpful options for formatting the output of our test runs. One of those options is –format usage. Using the “usage” format will show us which steps take the longest to run, how many times each step is executed and if any steps are unused. As shown in the usage report, I’ve slightly modified some of the example feature files provided with the Cucumber source to generate the following output.
Analyze
With our new data in hand, we can begin to zero in on the trouble spots. The usage report displays the steps ordered from slowest to fastest. The report also shows the number of times each step was executed. Your slowest step might not be the biggest problem. You should focus on the steps that run the slowest and the most often. Also, consider why the step was executed so frequently. If you wrote the scenarios following a behavior driven development (BDD) cycle, then the scenario was written to define behavior in the system. As a result, you may find many scenarios exercising the same part of the system in only slightly different ways. By viewing the scenarios in hindsight, you can consider if you should consolidate or eliminate some of the scenarios.
Improve
After this analysis, we now know where to focus our efforts. It’s time to make our improvements.
First is consolidation. In the “Analyze” phase, you considered if you could remove or consolidate scenarios. Now is the time to do that work. Deleting scenarios is fun and easy. Consolidating scenarios takes more effort and thought. Look for recurring patterns such as screen navigation and data setup. If you notice that you repeatedly set up the same data to run different scenarios, it’s a good bet that those scenarios are ripe for consolidation. From the regression testing perspective, you want to maximize the amount of testing performed while minimizing the amount of work you have to do in the system to prepare for the testing. Be careful to maintain the ability to independently execute the scenario.
Second, learn to wait. One of the most common causes of “slow downs” in cukes is “sleeping.” People often use sleep to wait for AJAX activity to complete. This is the most unpredictable and wasteful way to control test execution. Imagine that an AJAX action normally takes between .5 and 2 seconds to complete. If we sleep to wait for the action to complete how long should we sleep? Let’s say we decide to sleep for 3 seconds. If that action ever takes more than 3 seconds the test fails. In addition, every time the action takes less than 3 seconds the difference is wasted as it is not doing anything. Those seconds can quickly turn into minutes.
Third, learn to wait. Instead of sleeping you must learn to “wait.” The Selenium client includes a number of “wait_for” methods that will improve the predictability of your steps and will remove the waste caused by sleeping.
* wait_for_text (value) — Simply waits for the text you specify to appear in the DOM. Should this value already exist, execution will continue without waiting, so carefully select a value that you know won’t exist until the AJAX action completes.
* wait_for_element (xpathLocator) — Allows you to specify an exact xpath locator to wait for before continuing.
* wait_for_no_element (xpathLocator) and wait_for_no_text (value) — Allow you to wait for the removal of an xpath locator or text value from the DOM before continuing.
You can learn more about xpath and selenium from my Cucumber Pro Tips presentation, http://blip.tv/this-minute-in-programming/cucumber-pro-tips-the-secret-sauce-6078421.
Fourth, good software engineering. Once you’ve picked all the “low hanging fruit,” you’re left with your software engineering skills and training to profile. Tune and tweak the last bits of improvement from your steps.
Fifth, scenario tagging. Cucumber provides a simple yet powerful ability to add tags to scenarios using the @tagname. If you’ve made all the improvements that you can reasonably make but your tests still aren’t fast enough, then you can use tags to define a subset of scenarios to execute.
For instance, you could tag the scenarios with @smokeTest tag and run them by specifying ‘ –tag @smokeTest’ at the end of your cucumber command. Don’t forget to run your full suite of Cukes nightly, or better yet schedule them to run more frequently via your continuous integration server. Checkout the Tags page on the Cucumber wiki for more details on tags, https://github.com/cucumber/cucumber/wiki/Tags
Control
Congratulations. Give yourself a pat on the back. Hopefully you’ve met and surpassed your success criteria. Wait a second though; we’re not done. You need to keep an eye on things. Don’t let your test times creep up. That’s what the “Control” phase is all about and it goes on for as long as you care about the amount of time required to run the scenarios. Make the care and feeding of your cukes a regular part of your development cycle and they will provide your application with benefits for years to come.