An empirical study of user preference and performance with UML diagrams

Author(s): C. Britton, M. Kutar, S. Anthony, and T. Barker
Venue: Human Centric Computing Languages and Environments IEEE 2002 Symposia
Date: 3-6 Sept. 2002


This paper is a study on how user preference affects their effectiveness with different
types of UML diagrams. In this case, it was with regard to novices. I gather that the
idea was that they wanted to determine whether users that were involved in requirements
elicitation and validation but had no prior experience with UML would do best with the
diagram types that they preferred given minimal training or whether it would be best if
they gained a little experience before making such a decision.

For the experiment, they took 124 first year undergrad CSC students from the University
of Hertfordshire and Anglia Polytechnic in the UK who had little to no experience in UML.
I presume that the students did this experiment as a class assignment as opposed to
outright volunteering, but the paper didn't specify. Each student was given a questionnaire
with 6 scenarios on it. Each scenario was represented by a sequence or collaboration
diagram. There were five multiple choice questions for each scenario. The possible answers
for each were true, false, and can't tell. Prior to answering the questions, they were
asked which of the diagram types that they would prefer.

The results of the experiment were compiled from the questionnaires. Perhaps unsurprisingly --
given that these are novices who had little or no UML experience prior to this -- there
was no correlation between which diagrams that the students preferred prior to answering the
questions and correct answers. This does, of course, show that having stakeholders select
the types of UML diagrams that they'd like to use in requirements before they have actually
used the various diagram types is likely to result in them making poor choices. However,
it was found that when asked which diagrams that they preferred after answering the questions,
the students' responses did correspond to the correct answers. This would imply
that the exercise better informed the students as to which diagrams worked for them
and that they were thus able to adjust their preferences so that they actually corresponded
to what which diagram type was most effective for them.

All in all, I thought that the results were obvious enough that an actual experiment was
unnecessary, but I guess that they felt that experimental evidence was necessary. I can't
really say whether the results say anything positive or negative about UML comprehension,
but at least it gives some data about how the perceived usability of UML diagram types
by novices corresponds -- or in this case, doesn't correspond -- to the actual usability
of those diagrams.