Information Needs for Software Development Analytics

Author(s): Raymond P. L. Buse, Thomas Zimmermann
Venue: Software Engineering (ICSE), 2012 34th International Conference
Date: 2012

Type of Experiement: Survey/Multi-Case Study
Sample Size: 110
Class/Experience Level: Professional
Participant Selection: Random selection of participants were given the survey to respond to.
Data Collection Method: Survey


Many aspects of Software Engineering can be measured automatically, but prediction is still hard to do, making Software Development risky, so many Software Projects are delayed or never finished. Analytics can help developers and managers to make informed decisions based on data to reduce risk. There has been research done on the information needs of developers, but not managers. The paper focuses on the information needs of managers to achieve analytics that will help managers make project decisions.

The authors propose a study on the information needs of 110 developers and managers at Microsoft. They authors hypothesis that analytics can help answer important questions about software projects. They conducted a 28 question survey via email to determine the information needs for project decisions. The survey focused on identifying which factors were most important to decision making, having the participants rate them from not important to very important on a 4 point scale. They also rated the importance of different artifacts (test case, feature, etc.) on a 4 point scale. They were also asked which indicators they would use if they were made available, "failure information was ranked most important and bug reports second most overall".

The survey also had participants describe the scenarios in which analytics were useful and responses were summarized into main categories: targeting testing, targeting refactoring, release planning, understanding customers, judging stability, targeted training and targeting inspection. These categories are listed in order of most frequently mentioned.

Based on their results, the authors present a set of guidelines that any analytics tool should meet and identify analysis types that fit with their identified information needs. Analysis types are: trends, alerts, forecasting, summarization, overlays, goals, marketing, benchmarking and simulation. After their survey the authors explain that they believe analytics will help managers make informed software decisions and urge the community to develop new analytics tools.