A qualitative assessment of the efficacy of UML diagrams as a form of graphical documentation in aiding program understanding

Author(s): Scott Tilley and Shihong Huang
Venue: ACM Special Interest Group for Design of Communication, Proceedings of the 21st annual international conference on Documentation, SESSION: Understanding users II
Date: 2003


This paper "describes an experiment to assess the qualitative efficacy of UML diagrams in aiding
program understanding.” They wanted to see how well a group of software developers could
understand an existing program given only a set of UML diagrams for that program. The
experiment was conducted at a week long seminar on software architecture recovery and
modeling. This was a seminar that was by invitation only. Most of those attending were
PH.D. students or professors. All were experts in various area of software engineering and
all were extremely well versed in UML.

The participants for the experiment were volunteers from those attending the seminar.
There were 15 of them. They were given a 24-page document of a series of UML diagrams
on PerlCircle which is a complex web application written in Java by Rational Software
Corporation. The given one hour to fill out a questionnaire with 12 questions related to
likely changes to PerlCircle. It is unclear whether the participants were given the document
prior to the experiment or whether they had to read it in the hour as well, but given the
length of the document and that it was given with the consent form for the experiment
implies that they had time prior to examine the UML. Those doing the study read the
answers to the questions and drew conclusions from them for the results of their experiment.

While an experiment was obviously done, no real results are given. The experimenters
state a number of things about the efficacy of UML that are based on the results, but no
actual results are given. The decided that UML did help the participants understand the
program but with three main areas of difficulty: syntax and semantics, spatial layout, and
domain knowledge.

While standardized and commonly used, UML does have some ambiguities and the same
UML diagram does not always mean the same thing when written or read by different people.
This can cause difficulties in understanding UML diagrams.

The spatial layout can also have an impact on understanding. Poor layout can negatively
effect a developers ability to understand the diagrams. The participants were given the UML
diagrams on A4-sized paper with the diagrams in font as small as 6pt in some places. That
does not have a positive effect on comprehension of the diagrams but it can be difficult to
get large UML diagrams on a single sheet of paper or computer screen.

UML diagrams do not contain domain knowledge of the software that they model, so
if the developer reading them does not already have that knowledge, it can be difficult for
them to understand the diagrams properly. In the only actual piece of experimental data
given in the paper, one participant commented that they could not answer some of the more
challenging questions because they did not know enough about the server-side deployment
platform (J2EE).

All in all, the paper gives some useful information on the difficulties of understanding a
program given only the UML diagrams for it. However, while they did an experiment, they
did not give any of their direct results from it. There wasn’t even anything like 8 out of
12 participants could not answer question 7. All they really gave was a set of observations
about UML that could be easily drawn without an experiment. Having data back up those
observations would be useful, but any such data that they gained from the experiment
was not given. At least they actually attempted to do an experiment, and the paper was
informative, but their properly empirical results are lacking.

Average: 2 (2 votes)