“Talking tests”: a Preliminary Experimental Study on Fit User Acceptance Tests

Author(s): Marco Torchiano, Filippo Ricca, Massimiliano Di Penta
Venue: First International Symposium on Empirical Software Engineering and Measurement
Date: 2007

Type of Experiement: Controlled Experiment
Sample Size: 15
Class/Experience Level: Graduate Student
Participant Selection: In-class lab
Data Collection Method: Observation, Survey


This was a short paper, but nonetheless reports an experiment conducted to investigate the helpfulness of Fit test cases to clarify change requirements in a maintenance task. This paper emphasizes the importance of requirements in developing software with minimal defects. To test the impact of using Fit acceptance tests, the study participants were given four tasks of varying difficulty to implement on an existing software system. Two hypotheses were tested:

  • The availability of test cases does not significantly improve the capability of performing the maintenance task.
  • The availability of test cases does not significantly reduce the effort in the maintenance task.

After having the students work on adding the software tasks over two lab periods, the authors were able to reject the first hypothesis, that there is a statistically significant influence of the treatment on the capability to perform the tasks. The authors were not able to reject the second hypothesis. The authors were able to demonstrate fairly interesting results, that using Fit allows developers to finish more tasks, but more future research needs to be done to either confirm on contradict the preliminary findings.