What developers want and need from program analysis: An empirical study

Author(s): Maria Christakis, Christian Bird
Venue: Automated Software Engineering (ASE), 2016 31st IEEE/ACM International Conference
Date: 06 October 2016

Type of Experiement: Survey/Multi-Case Study
Sample Size: 375
Class/Experience Level: Professional
Participant Selection: Responded to Survey
Data Collection Method: Survey


This empirical evaluation is in regards to how developers responded to a survey about program analyzers. Some of the questions the study attempts to evaluate include: what barriers exist which hinder the adoption of program analyzers, what functional and non-functional characteristics do users want from program analyzers, and what code issues program analyzers should detect. After a preliminary review of the survey, a total of 375 developers responded to the survey with a response rate of 19%.

The results from the study indicated that some paint points for developers when using static analyzers included wrong default checks, false positives, and slowness. Additionally, developers would like static analyzers to detect security issues followed by best practices, concurrency, and performance. Some of the sources of unsoundness include exceptional control flow, aliasing, and arithmetic overflow. Moreover, the vast majority of users indicated they would like analysis results to be shown in the editor as opposed to a separate output. One set of users would like instant, superficial feedback while the other users would like a deeper analysis even if it meant the analyzer would take a few minutes. Finally, the live site analysis
of the study alluded to the fact that many developers might want program analyzers which test for the reliability of software.