: J. Spacco and W. PughVenue
: Companion to 21st. ACM SIGPLAN Conf. Object-Oriented Prog. Systems, Languages, and Applications (OOPSLA)Date
An important part of the software engineering life cycle is testing. Testing should be covered in a computer science curriculum. However, for students to really learn from it, they need to benefit from writing test cases in their own software. A big question is how do you teach it? Simply mandating testing by, say for example, 30% of your grade will be based on your test cases can be counter productive.
The University of Maryland developed a system called Marmoset for student programming project submission and testing. It requires that the professors provide skeleton code and a few test cases to get the student started. I feel however, that skeleton code takes away from the student’s design, so there is a little limitation there. The system allows unlimited submissions against public test cases, but limited access to more in depth test cases. By tracking submissions and test results, teachers can analyze what topics students are struggling with throughout the development of the project. Using this popular system, Pugh found that students tended to only perform the steps they were graded on. Attempts to change this were through limiting feedback of Marmoset, or by requiring testing, but even when required, students still just tested at the end. They did a study with CS2 students and two projects a MediaPlayer and binary search tree. The MediaPlayer group got meaningful feedback (by test naming conventions of what failed) from Marmoset, were the binary search tree group just got a yes or no for passing the test. The hypothesis was that students will write more tests in the binary search tree group since names are generic they couldn’t rely on it to find errors in their code.
The test between the MediaPlayer group and binary search tree group showed a statistically significant difference between the number of students achieving 80% code coverage before their programs were 100% functionally correct on the binary search tree project as there was for the media player project. One validity constraint of this was that the binary search tree project was assigned later in the semester, so the results could be entirely from students learning to write better tests later in the semester.
With such a small sample-size, conclusions can’t be made generic, however it is seen that without proper incentive to write test cases early, many students do not adopt a test-first mentality. They will stick to testing at the very end.