A Controlled Experiment Comparing the maintainability of Programs Designed with and without Design Patterns – A replication in a

Author(s): Marek Vokac, Walter Tichy, Dag I.K. Sjoberg, Erik Arisholm, Magne Aldrin
Venue: 2003 Luwer Academic Publishers
Date: 2003

Type of Experiement: Controlled Experiment
Sample Size: 44
Class/Experience Level: Professional
Data Collection Method: Survey


Length of study: 2 days
Number of Projects: 4
Type of Projects: open source
Language: C++
Size of Projects: small - medium
Independent Variables: programs, program version, change tasks, pattern knowledge
Dependent Variables: time, correctness
Design Patterns Analyzed: Abstract Factory, Composite, Decorator, Facade, Observer, Visitor

Threats to validity
Internal: group balancing, C++ proficiency, patterns experience, task attributes, domain knowledge

This experiment is replication of a previous experiment performed by Prechelt et al. comparing maintenance on software composed of designed patterns and on the software that doesn’t. However, this experiment aims for more realism by using professional software consultants and paying them for performing tasks during the experiment. Overall the new results still point to the fact that its hard to generalize the benefits or costs of software with design patterns, however it finds a lot of positive data that points towards benefits of patterns.

The experiment observed a general positive trend of improvement in time and correctness after the participants have taken a course in patterns than before. Maintenance with Observer and without was about equal in time. Decorator pattern allowed for solutions that were more correct that the non-pattern modifications. Comparison between software with Composite, and Abstract Factory also did not reveal any significant differences. Visitor pattern was the only pattern that seemed to have a negative impact, reducing correctness of solution. Most subjects did not even use the pattern, which indicates the difficulty of its comprehension.

I think this experiment is an improvement from the previous experiment, because of monetary reward for performing work. Thus there is not “educational” effect, and subjects are more likely to care about the quality of their solutions. However the fact that they knew that they were under experiment can be a significant factor in the results.