SEED: Software Engineering Evidence Database

Author(s): Kris Read, Grigori Melnik, Frank Maurer
Venue: Extreme Programming And Agile Processes in Software Engineering: 6th International Conference, XP 2005, Sheffield, UK, June 18-23, 2005; Proceedings
Date: 2005

Type of Experiement: Case Study
Sample Size: 45
Class/Experience Level: Undergraduate Student
Participant Selection: coursework
Data Collection Method: Observation

This paper described a study done with data gathered from two educational institutions for two different projects. In each case FIT was introduced as a required component of the project to be completed. Developers were given initial FIT test documents, were asked to ensure all tests passed, and asked to extend the test suite with new tests. Developers were formed into teams of 4-6 to complete the project. The following five hypotheses were tested:

1. No common patterns of ramp up or regression would be found between teams working on different projects in different contexts.
2. Teams will be unable to identify and correct “bugs” in the test data or create new tests to overcome those bugs (with or without client involvement).
3. When no external motivation is offered, teams will not refactor fixtures to properly delegate operations to business logic classes.
4. When no additional motivation is given, students will not continue to the practice of executing their tests in regression mode (after the assignment deadline).
5. Students will not use both suites and individual tests to organize/run their tests.

Several methods were used to gather data from the projects. A modified FitNesse binary was distributed to the developers which recorded information from each test-run, including the timestamp and number of failed and passed tests. Additional information was gathered by inspecting both the source and test fixture code. A subjective rating was assigned to each of the fixtures, looking at the “fatness” of that fixture, in addition to categorizing the different fixture types. After doing this, the authors were able to determine the more commonly used fixture types as well as which types produced the least favorable “fatness” fixtures.

In analyzing the results, the authors focused on the differences between the two courses. The courses differed in when FIT was introduced, in the beginning of one course and in the middle of another. The primary result of this was that started with FIT produced fixture code with a higher “fatness”, likely because all of the fixture code was written before any of the implementation code was written. The students also favored the simplest type of fixture code, resulting from the high learning curve of the FIT framework.