Expectations, outcomes, and challenges of modern code review

Author(s): Bacchelli, Alberto and Bird, Christian
Venue: 35th IEEE/ACM International Conference on Software Engineering (ICSE '13).
Date: 18-26 May 2013

Type of Experiement: Case Study
Sample Size: 1038
Participant Selection: Analysis of previous studies, Observations and interviews, Card sorting (grouped meetings) on interview data, Card sorting on code review comments, Creation of diagrams, Survey from managers and programmers
Data Collection Method: Observation, Survey, Project Artifact(s)


The case study was used to reaccess the current state of code reviews as of 2013. Originally, code reviews were in person, requiring programmers and managers to schedule time for the meeting. Nowadays, code reviews are more tool based, and in this study, Alberto and Christian observe those practices using CodeFlow as that tool for code reviews.

From their analysis, four main expectations emerged from managers and programmers going into a code review. Starting from greatest expectations to least: Finding defects in code, improving the code, alternative solutions to the code, and understanding and gaining knowledge to and from the code were the four main expectations. Other expectations included understanding code sources to further understand "big picture" of the project as well as training programmers to be more open and willing to share their code.

The actual outcome and expectation diverge however, as the results from the code reviews showed that code improvements are the most likely outcomes to occur versus a defect being found. Interestingly enough, another potential outcome, like the one I had previously with my Capstone, was knowledge transfer --- being told to read up certain external libraries for possible solutions to code.

The main reasons that code reviews switched from in-person meetings to tool based systems was time scheduling issues. Nowadays, the main challenges of a code reviews come from understanding the code. A huge plus for an author of the code is being able to divulge information about their code succinctly and clearly, so that code reviewers are able to offer suggestions and improvements in the deeper subtler avenues of the code (Subtle bugs, better improvements on data structure or design, etc) as well as being more willing to stand ground on their answers. Although tool based systems solve time scheduling issues and code reviews can be provided at pretty much any time of the day, the issue of non-synchronized meetings (reviewers are not reviewing at the same time for which the author is available) will hamper the reviewers in understanding the code.

Conclusions from the studies recommend to developers to have your own quality assured code before going in. This allows programmers to not worry about the more surface level issues of the code and more on the deeper, subtler ones. Code reviews are recommended to be given towards people who are similar in area: programmers who code review someone elses and are familiar with the structure overall will give better suggestions comparativ