A Large Scale Empirical Study on User-Centric Performance Analysis

Author(s): Zaman, S ; Adams, B ; Hassan, A.E.
Venue: 2012 IEEE Fifth International Conference on Software Testing, Verification, and Validation
Date: 17-21 April 2012

Type of Experiement: Survey/Multi-Case Study
Sample Size: 13
Class/Experience Level: Professional
Participant Selection: Ran tests to find the most popular use cases
Data Collection Method: Observation, Code Metric


Metrics that are used to analyze a system’s performance are generated by gathering results from a scenario from all users. A user’s perception of system performance should also be considered, but usually it is not. This empirical study analyzes the effect of performance on individual users to see if performance analysis results based on user perception is different from the scenario-based one.

The researchers created three case studies in which they conducted user-based and scenario-based performance analysis. User-based analysis measures performance from one user’s point of view whereas scenario-based analysis measures performance for all users. The three studies were on two commercial enterprise systems and one open source e-commerce system. The first case study produced four use cases that were feasible to test, the second case study produced six, and the last study provided three.

The authors found in the first enterprise system that the user-based analysis revealed serious performance problems that were not caught in the scenario-based metrics. In the second enterprise system, half of the users in each scenario had bad performance in at least 20% of the instances for user-based analysis whereas the scenario-based analysis showed less than that. In the e-commerce system, the user-based analysis showed that for each scenario, 40% of the users had bad performance in at least a third of the instances compared to the scenario-based one, which revealed numbers much less than that.

The user-based performance analysis in all three case studies showed users with severe performance issues in 10 out of 13 scenarios that were not visible in the scenario-based metrics. Based on the results from the study, the authors concluded that user-centered analysis should be conducted alongside scenario-based performance analysis.