Do Developers Read Compiler Error Messages?

Author(s): Titus Barik, Justin Smith, Kevin Lubick, Elisabeth Holmes, Jing Feng, Emerson Murphy-Hill, Chris Parnin
Venue: 2017 IEEE/ACM 39th International Conference on Software Engineering
Date: May 2017

Type of Experiement: Controlled Experiment
Sample Size: 56
Class/Experience Level: Undergraduate Student, Graduate Student
Participant Selection: students in software engineering courses at North Carolina State University
Data Collection Method: Observation


Modern integrated development environments usually have several design elements to support better understandability of error notifications. However, there has been limited research on understanding how developers perceive and comprehend error messages. Thus the researchers of this study conducted an eye tracking experiment with 56 participants to explain how developers make use of error message information to resolve defects within the IDE.

In the study, they focused on the investigation of three research questions and offered rationale for each. Some additional information of participants was also gathered, such as average years of professional software engineering experience, gender, mean age, and familiarity with Eclipse, the IDE being used for this study, and Java, the language being used. During the experiment, participants were asked to read 10 selected compiler error messages in random order and try to fix the error. They received no feedback on the correctness of their solution, and the average time to complete is about 45 minutes.

Screen recordings in video format and a time-indexed data file containing all eye movements were collected during the experiment, and they provided several materials on their website to support verification of the study. Some graphs and tables were created to analyze and show the results, and the study found that 1) participants read error messages and the difficulty of reading these messages is comparable to the difficulty of reading source code, 2) difficulty reading error messages significantly predicts participants’ task performance, and 3) participants allocate a substantial portion of their total task to reading error messages (13%–25%). These results offer empirical justification for the need to improve compiler error messages for developers.