Two Controlled Experiments Assessing the Usefulness of Design Pattern Documentation in Program Maintenance

Author(s): Lutz Prechelt, Barbara Unger-Maprecht, Michael Philippsen, Walter F. Tichy
Venue: IEEE Transactions On Software Engineering, Vol. 28, No. 6
Date: June 2002

Type of Experiement: Controlled Experiment
Sample Size: 96
Class/Experience Level: Undergraduate Student, Graduate Student, Professional


Time per task: 2 - 4 hours
Assessment of experience: knowledge test
Length of study: 6 hours total
Number of Projects: 2
Type of Projects: small non-comerical
Language: Java, C++
Size of Projects: small (360 LOC to 560 LOC)
Number of Classes: 6
Metrics: completion time, solution grade based on points, number of correct solutions
Independent Variables: order in which programs are given to students (pattern documentation, no pattern documentation)
Dependent Variables: completion time, number of correct solutions
Controlled Variables: are patterns documented,
Design Patterns Analyzed: Composite, Visitor, Observer, Template Method
Pattern Identification: patterns known before hand

Threats to validity
Internal: group balance, students tendency to give up
External: professionals can utilize comments better than students, real maintenance done in teams, programs are small and simple

Software Domain: String manipulation, GUI

These experiments determine whether pattern oriented maintenance is affected by additional documentation of design patterns. The two experiments are performed with two different groups of students at two different universities. They are set up so that the software which will be modified already has patterns in place, and maintainers already have pattern knowledge and experience (students had previous experience and were trained in addition). Maintainers perform pattern oriented changes, but they are not aware about the presence of patterns in the software. The experiments are testing two hypotheses: adding PCL (Pattern Comment Lines) improves completion time, adding PCL decreases errors in the completed tasks. Two small programs were used for maintenance tasks. Each came with regular documentation and extra Design Pattern documentation.

Experimental results on the first program were not very conclusive, however overall indicated that PCL usually leads to a faster solution with less errors. Completion time in one group for both PCL and non-PCL tasks were about the same, however that fact was counterbalanced by better quality of solutions for PCL tasks. Results from the second program do not indicate much discrepancy in the quality of solutions. However the PCL tasks were finished faster, with a possibility of up to 22 percent improvement.

I think that software projects that were being modified were relatively small, and the tasks not very complex. However since the assigned tasks still represented somewhat realistic maintenance tasks, the results in this study give a realistic indication of how well maintenances understand code. The size of the programs is sufficient for the study, because developers gain understanding of code in stages of relatively small coherent pieces. Also the fact that two different languages were used for each experiment increases the generality of the conclusions.