Testing Less Without Sacrificing Quality

When a new release is available, regression testing plays an important role in checking that changes do not negatively impact other parts of the software. The length of these tests might however conflict with the goal of having shorter release cycle. In their article, Kim Herzig, Michaela Greiler, Jacek Czerwonka and Brendan Murphy present a strategy that accelerates test processes without sacrificing product quality.

Software testing is viewed as a key activity to verify the quality of code. Working for Microsoft, Kim Herzig, Michaela Greiler, Jacek Czerwonka and Brendan Murphy had to find a way to improve the testing process for large Microsoft codebases like Windows, Office, and Dynamics.

To achieve this goal, they developed a cost based test selection strategy called THEO. This is a dynamic, self-adaptive test selection strategy that does not sacrifice software quality. THEO automatically skips test executions when the expected cost of running a test exceeds the expected cost of not running it. THEO is designed such that all tests will execute on all code changes at least once before the software release. Executing all tests at least once ensures that all bugs will be found. This way, software quality is not decreased, but the strategy might delay defect detection to later development phases. The underlying cost model is based on historic test measurements to estimate future test execution costs and causes no runtime overhead.

This model was evaluated by simulating historic development processes for three major Microsoft products: Windows, Office and Dynamics. These simulations covered more than 26 months of product development and more than 37 million test executions. This simulation showed a reduction of 50% of test executions and cutting down test time by up to 47%, saving millions of dollars per year, while maintaining product quality.

Read the full article on https://www.microsoft.com/en-us/research/wp-content/uploads/2015/05/The-Art-of-Testing-Less-without-Sacrificing-Quality.pdf