Work Experience versus Refactoring to Design Patterns: A Controlled Experiment

Author(s): T.H. Ng, S.C. Cheung, W.K. Chan, Y.T. Yu
Venue: Special Interest Group on Software Engineering (SIGSOFT)
Date: November 2006

Type of Experiement: Controlled Experiment
Sample Size: 118
Class/Experience Level: Undergraduate Student, Professional


Time per task: at least 3 hours
Length of study: 1 month
Number of Projects: 1
Type of Projects: Open Source
Language: Java
Size of Projects: medium (14,000 LOC)
Number of Classes: 207
Metrics: lines of code, years of experience, length of task completion, number of failed test cases
Independent Variables: maintenance's experience, direct and refactoring approaches
Dependent Variables: time to complete task, number of failed test cases
Controlled Variables: type of change tasks(perfective), language , IDE (Eclipse)
Design Patterns Analyzed: Adaptor, Command, Composite, Decorator, Factory Method, Mediator, Observer, Prototype, Singleton, State, Strategy, Template Method

Threats to validity
Construct Validity: refactoring done by authors
Internal: undiscovered design patterns, overlapping in tasks can make later tasks easier
External: one program with limited domains is used, only three change tasks

Software Domain: GUI, 2D graphics,

This experiment tests whether active refactoring with new patterns to accommodate future changes is more cost effective than continuing maintenance without refactoring. It is assumed that there are no patterns present which accommodate to newly planned changes in the current version of the software. JHotDraw is used as the experimental software system. Maintainers perform tasks on the original version and on the one refactored with patterns which accommodate changes. Authors hypothesize that there is a difference between performing direct maintenance versus maintenance with re factored system.

The results confirm the hypotheses proposed by the authors. The maintenance time is shown to be shorter for all tasks performed with refactored code in comparison to direct modification of unchanged code. This time represents the total time cost of performing the maintenance(for refactored approach time spent includes time that was used to refactor before maintenance). Both experienced and inexperienced developers benefited from refactoring approach. No significant difference in amount of defects was found between using refactored approach and direct approach.

Although this study included only 1 software system, the experiment is still relevant. The size of the system is a lot larger, and contains more patterns than systems in similar controlled experiments. Also the study had 118 participants and relatively complex change tasks. Such factors outweigh that fact that patterns are utilized in only one domain. However this study did not analyze the knowledge of patterns in their subjects, which can skew results even with a lot of regular professionals.