Code Reviews Do Not Find Bugs. How the Current Code Review Best Practice Slows Us Down

Author(s): Jacek Czerwonka, Michaela Greiler, Jack Tilford
Venue: International Conference on Software Testing
Date: 2015

Class/Experience Level: Professional
Participant Selection: Professionals at Microsoft
Data Collection Method: Observation, Code Metric


Code Reviews are a standard part of the modern software engineering workflow. They have many benefits, but exactly how effective are they, and can they be applied in a more productive way. We use code reviews to find defects, ensure code’s longterm maintainability, as a knowledge sharing tool, and to broadcast ongoing progress, but how best should we inject these code reviews into our workflow so that time spent getting people's opinions is justified. Through analyzing the outcome of code reviews at Microsoft, the authors looked more closely into how often code reviews resulted in finding functionality issues, if the skill-set of reviewers participating makes a difference, and how the social aspects of code reviews matter.

After looking at many different software projects, and their code review process, the authors learned that only 15% of comments indicated a possible defect, while over 50% were related to long-term code maintainability. In terms of the social aspects of the code review, people's role's on the team had a large effect on the outcome, as well as how familiar they were with the code base. The more code in the code base, the lower the overall rate of useful feedback. Modern code review is also very expensive, as it takes away from time a reviewer could be working on something new.