Fast Feedback Cycles in Empirical Software Engineering Research

Author(s): Antonio Vetro, Saahil Ognawala, Daniel Mendez Fernandez, Stefan Wagner
Venue: IEEE International Conference on Software Engineering
Date: 2015

Type of Experiement: Controlled Experiment
Sample Size: 1
Class/Experience Level: Professional
Participant Selection: Single group of stakeholders
Data Collection Method: Observation


This paper explores the consequences of introducing fast feedback cycles on empirical results of software engineering practices. Moreover, if fast feedback cycles can make empirical results of SE practices more valuable as it is noted traditional methods are time consuming and results are often outdated by immerging technology.

The researchers designed a proof of concept that incorporated an idea, contemporary with this paper. The idea of focus was hybrid Empirical Software Engineering [ESME], a scalable approach to empirical research “based on the combination of the cognitive power of manual hypothesis with the speed of automatic analysis”. The researchers developed a proof of concept with industrial partner and proponent of SCRUM. For the proof of concept the main objective was to discover “scope creep”, in other words whether a particular user story did or did not map to a project requirement. The instant feedback system created took the form of a generative model which used an inversion process called “Topic Modeling”, to learn the topics from a set of text documents (the user stories). With this feedback mechanism, stakeholders were asked were presented a select list of topics and asked to mark them with a specific non-functional or functional requirement. The generative model was fitted with this data and the process was repeated once again.

The researchers found that in the second iteration there was a significant increase topics marked with a non-functional/functional requirement than in the first iteration. The paper reports in the second iteration 23 non-functional/non-functional areas reported from 9 topics vs 5 areas from 3 topics in the first iteration. This research shows that it’s possible and efficient to capture domain knowledge through automatic analysis followed by fine tuning. In conclusion, the research suggests that fast automated mechanisms combined with data analysis can be useful for improving precision and in turn the usefulness of the analysis.