Assessing the Effect of Screen Mockups on the Comprehension of Functional Requirements

Author(s): F. Ricca, G. Scanniello, M. Torchiano, G. Reggio, E. Astesiano
Venue: ACM Transactions on Software Engineering and Methodology
Date: 2014

Type of Experiement: Controlled Experiment
Sample Size: 139
Class/Experience Level: Undergraduate Student, Graduate Student
Participant Selection: undergrad and graduate students both currently taking and not currently taking software engineering courses
Data Collection Method: Survey


When describing the desired functionality of a proposed system, the task of describing the functional requirements of the system can be tackled in many ways. A common methodology that has been adopted by the field is to write requirements as use cases. In conjunction with use cases, screen mockups are often included in requirements documentation to improve comprehension of the desired functionality of the system. The goal of this family of experiments was to quantitatively determine the extent of the benefits that adding mockups to use cases would have on the comprehension of the functional requirements of a system.

Over the course of Dec 2009 to Dec 2010, four replicated experiments were conducted on students from the University of Bascilicata, University of Genova, and Polytechnic University of Torino. Each experiment focused on a different set of students at the different universities in order to determine the generalizability of the results of the study. The sets of students included 2nd and 3rd year undergraduate Computer Science majors, 1st year graduate Math/Telecommunications majors, and 2nd year graduate computer engineering majors.

Each student in the experiment was tasked with comprehending a different set of use cases for two different systems. One set of use cases also came with a set of supplemental mockups. Students were given a set of 10 questions about the functionality of the system and had to provide answers that were designed to evaluate their comprehension of the system. After completing the survey for one system, the student was given a 30 minute break and then were asked to evaluate the second system in the same manner. In order to reduce threats to validity, the experimenters used a system called within-participants counterbalanced design which would place students in 4 different groups. Each group was designed to make sure that the order of the systems evaluated as well as the order of whether or not mockups were included in the first system or the second system did not affect the overall results of the experiments.

The results of the experiment evaluated the inclusion of mockups on different metrics - efficiency, effort, and effectiveness. Effectiveness was determined by calculating the accuracy of students’ answers on the comprehension surveys. Across all four experiment, the inclusion of mockups significantly improved the effectiveness of functional requirement comprehension. Effort was determined by the time it took students to complete the task. Across all four experiments, the inclusion of mockups did not improve effort of comprehension. Lastly, efficiency was determined by a formula that included the previous two metrics, directly proportional to effectiveness and indirectly proportional to effort. It was concluded that the inclusion of mockups was significant in improving efficiency in the comprehension of functional requirements.