Supporting Swift Reaction: Automatically Uncovering Performance Problems by Systematic Experiments

Author(s): Alexander Wert, Jens Happe, Lucia Happe
Venue: 2013 35th International Conference on Software Engineering (ICSE)
Date: 2013

Type of Experiement: Controlled Experiment
Sample Size: 12


This paper shows the application of Performance Problem Diagnostics (PDD) to performance problems in two e-Commerce systems. Since performance issues are a main concern when developing a software product Performance Problem Diagnostics could be useful to apply to industry problems. The removal of performance problems during integration becomes 21 to 78 times more expensive than fixing them during the design process. The later the problem is fixed, the more expensive it becomes. Already there are existing approaches to identifying performance problems by examining architecture, load tests, or runtime data. However, these approaches have their flaws.

In the experiment, 12 performance problems were defined for PDD to solve. The authors also evaluated if PDD can detect performance problems in actual industry applications. PDD was successful in identifying four performance problems in the TPC-W benchmark. The main idea behind PDD is that some performance problems have common symptoms and root causes. The authors had multiple detection strategies and formulas for PDD. After applying PDD to the poor performing systems, PDD identified four performance problems and increased the maximum throughput of the benchmark from 1800 requests per second to more than 3500. PDD has shown to be an effective and relatively easy method to improve the performance of a system.