UML collaboration diagram syntax: an empirical study of comprehension

Author(s): H.C. Purchase, L. Colpoys, M. McGill, and D. Carrington
Venue: First International Workshop on Visualizing Software for Understanding and Analysis
Date: 26 June 2002


Quality
3

Page(s):13 - 22

This paper was very similar to "UML class diagram syntax: an empirical study of comprehension."
It was done by more or less the same people at a later date. This paper focused on determining
which UML notations were more suitable with respect to human performance. This study looked
at collaboration diagrams. They looked at 5 different UML notations, each of which had two
variations with identical semantics. Participants were given a textual specification for the code
20 minutes prior to the experiment. They were then presented with sets of UML diagrams on a
computer in random order. They had to enter y or n, indicating whether the diagram matched the
textual specification. They had 60 seconds in which to answer or their answer was recorded
as an error. The diagrams were a mixture of the various 5 UML notations. The accuracy and
speed of participants in correctly understanding the diagrams were thus measured.

35 students were involved in this experiment. They were second and third year
Computer Science and Information Systems students at the University of Queensland who had
some experience with UML. They were given a tutorial on UML collaboration diagrams along
with the program specification 20 minutes prior to the experiment. The students were paid
$15 dollars but presumably volunteered for the experiment (it's not explicitly stated
how the students were obtained).

Of the two variants of each UML notation (labeled A and B), the experimenters chose the A
versions of the notations such that they were the ones that they thought were better
in terms of clarity and conciseness. The 5 varying notations weren't really compared with
one another. Instead, the focus appears to be on comparing the A and B varieties of each
notation. Personally, I think that this made it so that they were doing too much in
one experiment. They were really doing 5 experiments in 1. They were doing 5 experiments
comparing each A notation with its B variant. It does make it less obvious to the
participants what exactly is being tested, but it certainly doesn't reduce the number
of independent variables.

Interestingly enough, the results were inconclusive with regards to accuracy. Apparently
the numbers were close enough that the margin of error exceeded the differences. So,
in spite of the experimenters expectations, the A notations did not do significantly better
than the B notations. On the other hand, they weren't shown to be way off either since
the B notations weren't significantly better either. The speed results were about the
same with one exception. While 4 out of 5 of the notations had inconclusive data, the
5th one showed the A notation to be faster to comprehend.

This experiment might have given interesting data on differences between the notations
of UML collaboration diagrams but as a whole their data is too inconclusive to tell
us much.

0