Helping Analysts Trace Requirements: An Objective Look

Author(s): Hayes, J., Dekhtyar, A., Sundaram, S., Howard, S.
Venue: IEEE International Requirements Engineering Conference
Date: 2004

Type of Experiement: Controlled Experiment
Data Collection Method: Project Artifact(s)



This paper introduces many goals for an automated tracing tool and measures how well a research tool called RETRO satisfies some of these goals. The goals are based on the fact that a human analyst must use the tool. The goals are believability, which is composed of the subgoals accuracy, scalability, and utility; discernibility; and endurability.

Accuracy is measured as precision and recall. Discernibility measures good links rising to the top and bad links sinking using objective measures DiffAR and Lag. DiffAR is the difference between the average relevance of true links to the average relevance of false links. Lag is the average number of false positives with a higher relevance coefficient than a true link.

This paper measures the objective metrics mentioned above for a research tool (RETRO) built for the automation of candidate link generation. Recall was found to be 80.9%, precision 39.2%. DiffAR grows from 0.2 to .49-.76 and Lag drops from over 6 to less than 3. This shows that the algorithms used by RETRO meet the accuracy and discernibility goals of automated tracing tools.