Was it Worth the Hassle? Ten Years of Mobile HCI Research Discussions on Lab and Field Evaluations

Author(s): Jesper Kjeldskov, Mikael B. Skov
Venue: International Conference on Software Engineering
Date: September 2014

Type of Experiement: Survey/Multi-Case Study
Sample Size: 6
Class/Experience Level: Professional
Participant Selection: N/A
Data Collection Method: Observation, Survey


This paper observes the differences between the evaluation of mobile HCI in a lab study versus a field study. It particularly addresses the "is it worth the hassle?" question in a prior 2004 paper of the same authors regarding the efforts of field studies. It also summarizes the findings of other papers which cite this 2004 paper to further investigate the differences and benefits within lab studies and field studies.

After mobile HCI emerged as a valuable issue in software engineering research, its evaluation methodologies naturally followed. Thereafter, the distinction between lab and field studies has become important to conducting such evaluations. Lab studies are studies done in such a controlled environment for the purpose of research, whereas field studies are done in the user's environment wherein they would be using the evaluated application. The 2004 paper done by the same authors had observed and evaluated how many usability errors were detected by 6 nurses using an application in both field and lab. Results had shown that more usability defects had been found in the lab setting where more attention to detail can be focused on. This paper spurred many others to consider how a lab setting can save more time and be more effective by instead just observing a user's field (instead of interacting in it) to simulate as much of it in a lab. While many following papers have conflicting results (some confirming and others rejecting the 2004 paper), all agree that a field study takes twice, some even say three times, more longer than lab studies.

The authors conclude that after gathering 10 years of research on the differences of field versus lab study, the question may not be whether field study is "worth the hassle" or not. The authors present a different question while abandoning the one they have been asking. These papers emphasized a past challenge: what is usable design on a small screen, and is it worth it. We have come a long way to be confident in designing things on such size that maybe the question should now be "when" and "how" we should do it instead of questioning whether we should in the first place.