The Role of Experience and Ability in Comprehension Tasks Supported by UML Stereotypes

Author(s): Filippo Ricca, Massimiliano Di Penta, Marco Torchiano, Paolo Tonella, and Mariano Ceccato
Venue: Proceedings of the 29th International Conference on Software Engineering
Date: 2007


This paper is a summary of 3 identical experiments done by the authors of the paper. The
experiment in "An empirical study on the usefulness of Conallen's stereotypes in Web application comprehension"
is experiment II of this paper. The study attempted to determine the effects of Conallen's UML stereotypes on
comprehension of web-based applications. Conallen's stereotypes are a set of UML stereotypes designed with
web applications in mind. They add information on such things as navigation structure,
page generation, and form submission that UML diagrams do not normally contain explicitly. The
study was done on a 3 classes of software engineering students. Exp I was done with 13
grad students at the University of Trento, Italy. Exp II was also done at the University
of Trento but with 35 undergrad students. Exp was done at the University of Sanno, Italy
with 18 grad students. The UML diagrams in the experiment were derived from 2 pre-existing web
applications: Claros and WfMS.

Each experiment was done in two phases with 4 groups. Each phase was 2 hours long and it
is implied that they took place on separate days. The students were trained in Conallen's
notation prior to the experiment. In each phase, the students were given source code from
the application they were examining, UML diagrams derived from that code, and a questionnaire
of 12 questions. In the first phase, two groups got the code from Claros and two groups got the
code from WfMS. Of the groups with the same application, one's UML diagrams had Conallen's
stereotypes and the other's did not. In the second phase, each group switched applications
and UML type, so if they had been looking at Claros, they looked at WfMS (or vice-versa) and
if their UML diagrams had had stereotypes then now it didn't (or vice-versa). After each
lab, the students were given an additional questionnaire regarding the task, the system
complexity, whether they had had enough time to finish the lab, etc.

The various questionnaires were used to compile the results from the experiment. The results
were actually fairly interesting. While exp II showed that Conallen's UML stereotypes had
a large, positive effect on program comprehension, this was not the case with exp I and III.
In Exp I, there was not large comprehension difference between normal UML and UML with
Conallen's stereotypes. Exp II on the other hand, showed that there was a greater level of
comprehension with normal UML diagrams.

To explain the difference between the experiments, we need to look at the students' ability
scores. The students were ranked according to ability (based primarily on grades) -- in part
to be able to balance out the groups in each experiment with regards to ability. It turns
out that the undergrads with high ability and the grads generally did not derive much additional
benefit from the Conallen stereotypes. It was the low ability undergrads that benefited
from them. The authors also found that the grad students and high ability undergrads spent more
time reading the code than the diagrams. They made three main observations from this: