Automatic Summarization of Bugs

Author(s): Sarah Rastkar, Gail C Murphy, Gabriel Murray
Venue: IEEE Transactions on Software Engineering, volume 40 issue 4 (30 November 2013), pages 366-380
Date: 2014


Repositories are useful for learning details about a project because they contain a huge amount of information. Software developers typically have to analyze bug reports to better understand code and how and why changes were made to the code. In order to speed up the process of having to read a large amount of bug fixes and changes, this article introduces an automated summarizer to provide shorter summaries of bug reports. The goal was to allow developers to peruse the repository and efficiently despite some reports lacking sufficient details and short summaries.

In order to judge the effectiveness of the summaries created by the summarizer, a study was performed. The summarizer created 3 summaries for each bug using different classifiers: email data, email and meeting data, and bug report data. These summaries were first tested to ensure they were good summaries. Next, the classifiers were compared to one another to see which was most effective. This was done using a variety of calculations including F-score, recall, precision, and pyramid precision. The performance was also tested using 6 paired t-tests which found that the bug report classifier generated better summaries than the other 2 classifiers with statistically significance. This found that the most effective classifier was the bug report classifier. Lastly, a test with human participants was conducted in which the participants were required to perform a set of 8 tasks and were measured on both satisfaction and time to completion. On average, tasks took 8.21 minutes to complete with the longer documents while the summaries only took 5.21 minutes. In terms of satisfaction, participants 9 of the 12 participants thought the summaries were easier to read, clearer, and less intimidating. The other 3 would have liked the originals on hand to verify their tasks. Thus this study was able to show that an automatic summarizer is feasible and allows for developers to more efficiently read through bug reports.