An empirical investigation of capture and access for software requirements activities

Author(s): Heather Richter, Chris Miller, Gregory D. Abowd, Idris Hsi
Venue: GI '05: Proceedings of Graphics Interface 2005
Date: May 2005

Type of Experiement: Case Study
Sample Size: 12
Class/Experience Level: Graduate Student
Participant Selection: classwork, random selection
Data Collection Method: Observation, Project Artifact(s)

Quality
3

The paper explores how to to benefit from recorded meetings through the capture and access of Knowledge Acquisition (KA) sessions. The author discusses how the sessions would be used in the creation of a software requirements document. The paper evaluates the access of of recorded and indexed KAKA sessions in comparison to just simple recordings. The authors discusses a prototype called TagViewer that "allowed participants to use semantically meaningful indices to browse, search, and find relevant information in the transcript and video." They also noted that the "purpose of capturing and indexing these sessions is to provide the KA team members, and later software developers, the means to find and review portions of the original KA conversation as needed." The paper concludes that although the "use of TagViewer in this experiment did not yield strong quantitative benefits, there were very clear differences in the use of the video record that provide insight into the potential of such capture tools to improves access to important discussions."
Link: http://portal.acm.org/citation.cfm?id=1089508.1089529

How long was the study? One hour total session time
What software or prototypes did the study used? TagViewer, an interface prototype allowing users to view all of the tags of a knowledge acquisition session to index into the tagged transcript and video. TAGGER (Team-Aware Acquisition Guide for Goals, Entities, and Relationships), a tool to capture and record knowledge acquisition sessions, and enable tagging of that recording in real time.

Process Outline

  1. KA sessions were between one of the authors and two airline pilots
  2. Subjects elicit "take off procedures and checklists with the purpose of designing an electronic checklist system for the cockpit"
  3. Subjects using TagViewer were "given a brief training session on the tagging process and told they would have a tagged transcript along with the video"
  4. Subjects were given a "simplified requirements framework of five categories" but can use any other if they chose to
  5. Subjects were asked to fill out the requirements document in 45 minutes

RESULTS
Requirements Quality (Items Identified): No real difference between the two conditions in the number of items identified
Requirements Quality (Errors Identified): There is a difference (p-value of 0.12) in the number of errors made
Requirements Quality (Overall): Does appear that using TagViewer may improve the accuracy of the requirements document
Behavioral Results: Participants using TagViewer spent considerably less time looking at their notes and at their requirements than their counterparts with more focus on TagViewer. Several of the participants using TagViewer had less than one page of notes.

0