An Empirical Study on the Impact of Automation on the Requirements Analysis Process

Author(s): Lami, Giuseppe, and Robert Ferguson
Venue: Journel of Computer Science and Technology Thursday
Date: 2007

Type of Experiement: Case Study
Class/Experience Level: Professional


The paper discusses how the automation tool, QuARS, is "used in a formal empirical experiment to assess the impact in terms of effectiveness and efficacy of the automation in the requirements review process of a software company." “QuARS lexically and syntactically parses the textual requirements in order to identify any potential ambiguity.” The quality models used for requirements analysis are expressiveness, consistency, and completeness. “The aim of the study described in this paper is to evaluate the effects of the use of QuARS for analyzing NL requirements in a real industrial project.” The paper concludes that the automatic tool QuARS can significantly improve the requirements analysis process.

What were the subjects? Requirements written under NL (Natural Language) in a real industrial project
What software or prototypes did the study used? QuARS (Quality Analyzer for Requirements Specification)

Process Outline

  1. Company performs analysis of NL requirements by assigning external consultants to review the documents in order to identify and report defects
  2. Company collects performance data of the requirements analysis process as the effort required and the number of defects found
  3. Run QuARS on the documents reviewed by consultants
  4. Outcomes in terms of defects found and effort required were compared with the analysis reports produced by the consultant


  1. Hypothesis 1: The average effort per defect found with QuARS is less than that with human inspection
  2. Hypothesis 2: The number of defects detected with QuARS is higher than that with human inspection
  3. Hypothesis 3: The Symmetric Difference of the set of defects found by QuARTS and the set of those found by the human inspector is not null
  4. Hypothesis 4: The effort to identify false positives is higher than that to identify actual defects
  5. There is also other data and results that have not been displayed here.