Are there any Unit Tests? An Empirical Study on Unit Testing in Open Source Python Projects

Author(s): Fabian Trautsch, Jens Grabowski
Venue: 10th IEEE International Conference on Software Testing, Verification and Validation
Date: 18 May 2017

Type of Experiement: Case Study
Sample Size: 10
Class/Experience Level: Professional
Participant Selection: Open Source Python Projects
Data Collection Method: Code Metric, Project Artifact(s)


This paper present an empirical study of four aspects of Unit Testing. The study investigates if the developers are correctly categorizing their tests as unit test, the proportion of real unit test within these projects, if mock are being used and how, and how the number of unit test evolve over time. The researcher conducted a study which analyzed 10 random open source Python projects taken from GitHub which had over 200 revision.

From the investigation, researcher came to a conclusion for each of the four aspects. First, when compared to the ISTQB/IEEE unit test definition, the developers where not correctly classifying unit test. Rather, they were over classifying test as unit test. Secondly, of the correctly classified unit test, IEEE unit test were on average proportionally higher than ISTQB. Third, the findings of the study show a wide range of the level of use for mocks. The research were unable to conclude a consistent use or lack of use across the projects. Finally, the researcher were able to show four patterns for number of unit test as revision progress: beginning raising to stagnation, beginning raising to drop, constantly raising, constantly low.