Impact of CS programs on the quality of test cases generation: an empirical study

Author(s): O. Gomez, S. Vegas, N. Juristo
Venue: ICSE '16 Proceedings of the 38th International Conference on Software Engineering Companion
Date: 2016

Type of Experiement: Controlled Experiment
Sample Size: 127
Class/Experience Level: Undergraduate Student, Graduate Student, Professional
Participant Selection: Graduate and undergraduate students.
Data Collection Method: Observation, Code Metric


The purpose of this paper is to find out whether or not computer science programs have an impact on the quality of test cases generated. In this case, quality means the number of failures the test cases report; the test case quality is higher when they reveal more failures. There have been several studies in the past to indicate that formal software training is deficient. Other studies purposefully inject code with bugs and had students write test cases to see how many failures they could find; in this case, researchers used a metric/formula to determine the effectiveness of each test case.

The research question addressed within this article is as stated: "Is the level of CS program exposure related to the quality of test cases generated with black-box and white-box methods by undergraduate and graduate students?" Once again, the "quality of the test cases is measured as a percentage of the number of faults revealed by the generated test cases to the total number of faults injected in an instrumented program." In total, four experiments were run where randomly selected graduate/undergraduate students would test using strictly black or white box testing on a faulty "ntree" program.

The results from these experiments are as follows:

- "The probabilities of generating test cases that reveal failures differ significantly depending on the level of exposure to CS knowledge."
- "There is found to be no significant difference generating test cases using either white-box or black-box."
- "The probabilities of generating test cases that reveal failures against test cases that do not reveal failures are less at lower levels of exposure to CS knowledge."
- "Students who have had less exposure to CS knowledge generate poorer quality test cases."

Overall, they find that students should have more exposure to testing; whether this be incorporated within current courses, or added as separate courses entirely. They suggest that students should be exposed to testing early within the course curriculum and gradually expand their knowledge within the area as they go through more advanced courses.