Usage, costs, and benefits of continuous integration in open-source projects

Author(s): Michael Hilton, Timothy Tunnel, Kai Huang, Darko Marinov, Danny Dig
Venue: International Conference on Automated Software Engineering
Date: 2016

Type of Experiement: Survey/Multi-Case Study
Sample Size: 442
Class/Experience Level: Professional
Participant Selection: responded to survey
Data Collection Method: Observation, Survey, Project Artifact(s)


This study aims to collect empirical data on the usage of Continuous Integration within the professional world, a topic that the authors claim has rarely been studied before. They are the first to collect data on hundreds of open-source projects related to the use of CI as well as survey the opinions of over 400 open-source developers about why or why not CI is used on certain projects. The sample size is large enough to reflect open-source developers as a whole, but it would have been interesting to see a comparison of the usage of CI with for-profit software projects.

Most of the primary data for the study is scraped off of GitHub where information about an open-source project and it's CI solution is usually publicly available. Data about the usage of CI reveals that Travis CI is the most popular service by far. Of the 34,544 projects analyzed on GitHub, 40.27% use CI, 90.1% of those use Travis, and that the more popular the project on GitHub, the more likely it was to use CI. There is additional data comparing CI usage to the programming languages used in the project and the number of projects over time that adopt CI.

The survey of 442 developers gives insight into the costs and benefits of CI. A clear majority of about 90% responded that they will most likely use CI for their next project. The top reasons for doing so are that CI makes developers less worried about breaking builds and helps catch bugs early on. The primary reason for not using CI is the lack of familiarity with CI among team members and the lack of automated tests. Other observations made are about build times, build successes and failures, and frequency of updating CI configurations. In conclusion, CI is on the rise and rightly so considering the benefits it offers. There is still little research on the subject, but the growing trend is expected to continue and more research will greatly improve how it will be used in the future.