A Controlled Experiment in Maintenance Comparing Design Patterns to Simpler Solutions

Author(s): Lutz Prechelt, Barbara Unger, Walter F. Tichy, Pater Brossler, Lawrecne G. Votta
Venue: IEEE Transactions on Software Engineering
Date: December 2001

Type of Experiement: Controlled Experiment
Sample Size: 29
Class/Experience Level: Professional
Data Collection Method: Survey


Team Size: 6 - 8
Time per task: 3 hours max
Length of study: 2 days
Size of Projects: small - medium
Number of Projects: 4
Language: Java

Independent Variables: programs, program version, change tasks, pattern knowlege
Dependent Variables: time, correctness
Design Patterns Analyzed: Abstract Factory, Composite, Decorator, Facade, Observer, Visitor

Threats to validity
Internal: team size, team experience, accuracy of time stamps
External: knowledge/experience with patterns, amount of documentation, not all patterns tested

This study attempts to understand just how much effect design patterns have on software maintenance. Specifically, design patterns are compared to simpler solutions and whether there is significant improvement in time, size of changes, and program comprehension in one approach over the other. Four different software software packages are used to perform maintenance, and each of them comes in two versions – one that doesn't use design patterns and one that does. For each package two maintenance tasks are assigned, and they involve modifying different parts of each product and require understanding different patterns. The experiment is performed with a group of 29 professional software engineers. Since design patterns are not well known, maintenance is performed after subjects are taught patterns and before. Before experiments are performed authors hypothesis that adding more complexity with design patterns to simpler solutions will be harmful in the process of maintenance.

The results of the experiments show that Design Patterns can both improve and hurt maintenance. The decorator pattern proved to be easier to modify because of the separation of functionality. The Visitor, Abstract Factory, and Composite did not have much of a difference in performance as opposed to the simpler solution. It took longer to perform a maintenance task on Observer than a simple solution, because of the extra complexity it brought. Since most of the patterns tested at either improved the maintenance time or kept it even with alternatives, the paper concludes it is overall a good idea to choose patterns when designing software. The extra complexity of patterns makes the software less coupled as more and more changes are applied while keeping the leaving the maintenance time the same, as opposed to simpler solutions which couple the software further. Results also show that software engineers aware of patterns perform maintenance faster than those who are not. The paper concludes that this may be problematic when the amount of patterns increases, and thus each developer needs to learn more in order to be effective.

I think the study had a good setup. The teams were selected randomly and fairly, taking into account the different experience level of the developers. Also i think it was a good idea to give the developers training in patterns and experiment with them before and after the training. This showed how much effort is needed by untrained maintainers to understand complex designs. However unclear assignment was given for one of the tasks involving the visitor pattern and not enough data was gathered about it.