Performance Analysis for Object-Oriented Software: A Systematic Mapping

Author(s): David Maplesden, Ewan Tempero, John Hosking, John Grundy
Venue: IEEE Transactions on Software Engineering
Date: 2015

Type of Experiement: Survey/Multi-Case Study
Class/Experience Level: Professional
Participant Selection: Empirical Performance Analysis papers later than 2000
Data Collection Method: Survey


Object-Oriented programming is used heavily in industry today, and this makes the performance of OO software particularly interesting to Software Engineers and researchers. However, due to the growing scale of software and the complexities of OO programming, it is not easy to analyze the performance of software. Object Oriented programming can lead to a greater number of methods, and its focus on reusability and flexibility leads to extremely layered and complex systems. This layering and complexity causes what the article calls “runtime bloat,” where excessive activity (method calls and layers) is necessary to achieve a seemingly simple task. In order to find performance analysis techniques to apply to OO software, the writers conduct a survey of empirical performance analysis techniques found in literature starting in 2000.

When conducting the survey, the writers decide to focus on empirical performance analysis, because of its prevalence in industry and their desire to keep the survey a manageable size. Empirical performance analysis is one of two approaches to Software Performance Engineering, where the analysis is taken from data measurements from running software. The other approach is predictive model-based performance engineering, where a model is built of a software system in order to predict its performance behavior, and it is not included in this survey. The process for the survey is as follows: 1) Define research questions. 2) Perform manual searches to pilot inclusion/exclusion criteria. 3) Develop automated search strategy. 4) Formalize review protocol. 5) Perform search of articles. 6) Data extraction. 7) Analysis.

After searching and excluding articles, 253 papers were included in their survey, and data was extracted from each one for analysis. The papers were evaluated against the chosen research questions, and the paper details the results from each question. The results from the first question, “What approaches to empirical performance analysis have been proposed that are applicable to OO software,” were three groups of papers: Profiling, performance analysis, and profiling infrastructure. The second question, “How can these approaches be characterized,” resulted in several characteristics: Venue, Domain, Focus, and Analysis Type. Question 3, “What form of feedback does the approach provide,” found 53 papers that provided some sort of feedback that could be analyzed. Research question four was “How practical to apply are the approaches?” The researchers found that 39% of the approaches relied on dynamic binary instrumentation and were straightforward to put to use, and another 14% relied on data that could be gathered by any suitable performance monitor, thus making them easy to use. The rest of the papers outlined techniques that were not practical to apply, so approximately 53% of the papers in their limited search outlined practical performance measuring techniques. Finally, question 5 asks “How are the approaches evaluated and validated?” They found that approximately 31% did no validation, and only 1.6% did actual validation with user studies.

After analysis was complete, the paper comes to 4 conclusions. First, there are few approaches that use in-depth static information or analysis when analyzing performance. The next thing that they found was that the research was across many different domains. The third thing they found was that there is not much research into the utility of performance analysis approaches, especially for large-scale object-oriented software. Finally, they found that there is a lot of room to improve in feedback from performance analysis on large-scale projects.