A Large-Scale Empirical Study of Just-in-Time Quality Assurance

Author(s): Yasutaka Kamei, Emad Shihab, Bram Adams, Ahmed E. Hassan, Audris Mockus, Anand Sinha, and Naoyasu Ubayashi,
Venue: IEEE Transactions on Software Engineering Vol. 39, No. 6
Date: June 2013

Type of Experiement: Case Study
Sample Size: 14
Class/Experience Level: Graduate Student, Professional
Participant Selection: Projects for experimentation were selected based on availability, popularity, and scale .
Data Collection Method: Observation, Code Metric, Project Artifact(s)


Quality assurance (QA) is important, but finding the balance between cutting costs and not having enough QA processes in place is difficult. In the past, defect prediction models have been used to determine which components or modules could contain the highest number of defects. While there are a few reasons as to why this causes problems, the largest factor is the predictions are made late in the development cycle. In order to prevent this, change-level or “Just-In-Time” (JIT) prediction serves as a method to catch QA issues at the time changes are submitted, which means those who submit changes are automatically linked to defects they will most likely be able to understand. The authors looked at 11 projects (6 open source and 5 commercial) to determine if JIT prediction would answer three questions:
1. How well can JIT predict defect-inducing changes?
2. Does prioritizing changes based on predicted risk reduce review effort?
3. What are the major characteristics of defect-inducing changes?

The authors looked at 14 factors to determine whether or not a change in the projects’ code would lead to a defect. After testing questions one, it was determined predicting defect-inducing changes is effective for both open source and commercial projects with the average accuracy being 68% and the precision being 34%. For question two, 20% of the effort can detect an average of 35% of defect-inducing changes, but this is only true when using an effort-aware prediction model. For question 3, out of the 14 factors tested, the number of modified files and whether or not the change is a defect fix are considered risk-increasing and the average amount of time between the last and current changes are risk-reducing. Overall, JIT does help in minimizing effort by highlighting potentially risky changes early and reduce the amount of money needed to produce software.