Automatically Generated Acceptance Test: A Software Reliability Experiment

Author(s): Peter W. Protzel
Venue: Proceedings of the Second Workshop on Software Testing, Verification, and Analysis
Date: 1988

Type of Experiement: Case Study
Class/Experience Level: Professional


This older paper presents a study in which automatically generated acceptance tests were applied to a project as the NASA Langley Research Center. This research specifically attempts to identify the number of faults acceptance testing uncovers or prevents. The actually application used in this study is a classic example used in various research publications, but is rather outdated for modern comparison. This paper ended up being very misleading and not too valuable in practice.

While this study did automatically generate acceptance tests, the type of tests generated needs to be explained. Without getting into too much detail, the computer system uses bit patterns to represent different states. This study created acceptance tests by systematically testing against all possible bit patterns and then consulting an oracle. While the authors call these acceptance tests, and they may be, they are definitely not user-acceptance tests and therefore should not really be called acceptance tests.