Empirical Evaluation of the Fault-detection Effectiveness of Smoke Regression

Author(s): Atif M. Memon and Qing Xie
Venue: 20th IEEE International Conference on Software Maintenance
Date: 2004

Type of Experiement: Controlled Experiment
Sample Size: 4


This paper describes a technique for regression testing a GUI-based software system. While this paper is not directly tied to acceptance-test driven development, it very much could be used to augment initial acceptance tests as they are transitions to automated regression tests. This paper aims at attempting to identify the useful characteristics of effective regression testing of GUI applications.

To facilitate this research, the authors created smoke tests for four different applications. 200 variations of each of these applications were created with random seeding of faults. 5000-8000 smoke test cases were created to test the GUI application, for a total of 4 million test runs. The authors were able to generate four contributions to regression smoke testing of GUI applications:

  • A first empirical study evaluating the strengths and weaknesses of GUI smoke tests.
  • Five new test oracles for GUI smoke tests, their relative strengths and costs.
  • Relationship between GUI smoke tests and coverage of the underlying code.
  • Classifcation of faults that can and cannot be detected by GUI smoke tests.

The regression smoke testing was fairly successful, with 60% of faults detected. The weakness in this fault detection was not the actual smoke tests, but the testing oracle. Faults seeded in dead code also were not caught with the smoke testing. Code coverage was also fairly good with this testing style, with roughly 50-70% coverage. Overall, the authors conclude that is a very promising technique and can be a very valuable and relatively cheap alternative to not testing in the long run.