An Automated Approach for Goal-driven, Specifcation-based Testing

Author(s): Kristina Winbladh, Thomas A. Alspaugh, Hadar Ziv, and Debra J. Richardson
Venue: 21st IEEE/ACM International Conference on Automated Software Engineering (ASE '06).
Date: 2006

Type of Experiement: Case Study
Sample Size: 2
Data Collection Method: Observation


This paper presents some work related to acceptance testing, namely specification-based testing. This testing attempts to validate that a software system directly behaves as specified; acceptance testing verifies a system behaves as the end-user expects. The end-user could specify acceptance criteria in the specification and in doing so both techniques would be the same. This specification-based technique works by annotating the code with additional information, including pre and post conditions.

The first case-study was performed on a small academic application. The second case-study was performed on a larger, industry-level application. In both these cases, this specification-based testing compared goals and plans against the actual program code. In the smaller case-study the authors found that the testing did good finding both false-positives and domain knowledge errors. In the larger case-study, however, the technique did not produce as valuable results. The authors conclude that this is a result of the difficulty in creating a robust oracle. Regardless, these findings are not very promising at all in terms of usefulness, as real-world projects are often times many-factors larger and complex than simple academic examples.