The Design Space of Bug Fixes and How Developers Navigate It

Author(s): Emerson Murphy-Hill, Thomas Zimmermann, Christian Bird, and Nachiappan Nagappan
Date: JANUARY 2015

Type of Experiement: Survey/Multi-Case Study
Sample Size: 403
Class/Experience Level: Professional
Participant Selection: By having the first author go to a building that housed a particular product group, Attended six bug triage meetings across four product groups, Sent the survey to 2000 randomly selected recipients from a pool of all employees of Microsoft who had “development” in their job title, and were not interns or contractors
Data Collection Method: Observation, Survey


The authors of Design Space of Bug Fixes and How Developers Navigate It (“Design Space”) utilize a mixed-method study composed of motivating case study, qualitative interviews, data collection from bug triage meetings, and a survey to investigate factors that influence how bugs are fixed. Based on the data gathered and analyzed via these methods, the authors discuss implications such as how to make bug prediction and localization more accurate.

After presenting the background and related work on the topic of Design Space, the authors present a motivation for the rest of the study with a case study conducted with a browser-based learning environment, Pex4Fun. They illustrate the motivation with actual code examples submitted by the users of Pex4Fun, and pose the following questions: (1) What are the different ways that bugs can be fixed? (2) What factors influence which fix an engineer chooses?

Given the question from the aforementioned motivation, the authors provide details on their “mixed” methodology, which they explain lets them study the research questions in as broad a way as possible and triangulate the answers to improve accuracy. The authors illustrate five mixed-methods that were utilized in detail.

The first and second method, Opportunistic and Firehouse Interviews, were conducted by visiting a building that housed a particular product group and asking a series of questions regarding the last bug that the engineers in the building were involved in fixing. The goal was obtain qualitative answers regarding their bug fix design decisions.

The third method, Triage Meetings, were conducted by attending different product groups’ bug triage meetings and observing the engineers and taking notes. The goal was to obtain answers to the authors’ research questions with respect to how engineers work together to find good bug fix designs.

The last two methods, Survey and Replicated Survey, were put in to quantify the authors’ observations made during the interviews and triage meetings and replicate the quantified findings outside of the initial subject group. The surveys were posted in different contexts with different surface terms, but contained replicated data in both instances.

Aggregating the analysis of these methods, the authors of Design Space conclude that the design space of bug fixes is multi-dimensional, and that engineers navigate the space by, for example, selecting the fix that is least disruptive when a release looms near. In addition, the authors list a few topics that the outcome of their study implies. Those topics are regarding: Additional factors in bug prediction and localization, Limits of bug prediction and localization, Refactoring while fixing bugs, Usage analytics, Utility analytics, and Fix reconsideration.