The Art of Testing Less without Sacrificing Quality

Author(s): Kim Herzig, Michaela Greiler, Jacek Czerwonka, Brendan Murphy
Venue: IEEE International Conference on Software Engineering
Date: 2015

Type of Experiement: Other
Sample Size: 3
Class/Experience Level: Professional
Participant Selection: Microsoft Product Development
Data Collection Method: Observation, Code Metric, Project Artifact(s)


Whenever a large product or system makes changes, testing plays a critical role in product quality. However, a lot of the time testing is time consuming, expensive, and rarely find defects. Thankfully, Microsoft has developed a generic test selection strategy model called THEO that accelerates test processes without sacrificing product quality. The strategy model is a cost model that calculates whether or not a test is valuable enough to run. THEO automatically skips test executions when the expected costs of running a test exceeds the cost of not running it at all. The goal of TEO is to decrease development cost and increase productivity without sacrificing product quality.

The theory is that a test cost is evaluated using several factors using data feeds and history of the test case. Eventually, a test will have been run multiple times and cost a certain amount of time to run. If that test cost exceeds the cost of it not being tested at all, then that test is skipped. For defects that are accidentally skipped, there is a cost function that allows Microsoft to minimize costs and speed up development for incorrectly skipped defects. THEO does not run test cases, but rather decides which test cases are better to run and which add more value to the product. Theoretically, THEO reduces the number of test executions by up to 50% and test time by up to 47% and does not sacrifice product quality. The 0.2% to 13% of defects that were skipped would cost more, however, the benefits of skipping the costly tests outweighed the added cost. The author suggests these techniques be applied to live production test teams in all product teams.