Code Review Quality: How Developers See It

Author(s): Oleksii Kononenko, Olga Baysal, Michael W. Godfrey
Venue: Internation Conference on Software Engineering
Date: 2016

Type of Experiement: Survey/Multi-Case Study
Sample Size: 88
Class/Experience Level: Professional
Participant Selection: Mozilla core developers who responded to survey
Data Collection Method: Survey


The study aims to shed light on the factors that impact code reviews and their quality. Ideally, code reviews are done in a thorough fashion, where time is taken out of the developer's and reviewers' day to discuss the patch. In the study, the authors created a survey for their participants in order to gather information as to how they felt about the code review process as developers and whether or not it fulfilled it's purpose effectively.

From the survey, they discovered the developers opinions on what factors affected each part of a code review, such as the time taken and how decisions are made. They also asked questions about what affects their perception of the patch quality, what makes a good patch review, and what factors would affect the quality of the code review.

The results showed that for the developers the top factors in how long a review would take were based around the size of the patch, the experience of the developer and the number of files altered by the patch. Decisions made during the code review would usually be made around the writer experience and the reviewer experience. Their perception of the patch quality was mainly affected by the experience of the writer as well as whether or not coding conventions were followed. The code review quality is affected mainly by the code reviewer experience. Seeing the results of their survey the authors discussed the possible directions following their research. For example they believe that a system for recommending reviewers would be most effective, given how much of the process is affected by the experience of the reviewers, as well as ensuring that those who weren't as experienced would be able to start out with easier code reviews to build experience as needed.

To conclude, code reviews are processes that are far from being at their most efficient, decision-making and time consumed is wrought with the biases about the writers and the reviewers. Quality of the feedback given is heavily influenced by the experience of the reviewers and their perception of the code quality.