TestFul: an Evolutionary Test Approach for Java

Author(s): Luciano Baresi, Pier Luca Lanzi, and Matteo Miraz
Venue: 2010 Third International Conference on Software Testing, Verification and Validation
Date: 2010

Type of Experiement: Quasi-Controlled Experiment
Sample Size: 4
Class/Experience Level: Undergraduate Student
Data Collection Method: Observation, Code Metric


The paper introduces a new evolutionary testing method, TestFul, that automatically generates unit tests for Java classes. The article explains the details and background of the algorithms used to implement TestFul and reviews other automated test approaches and frameworks. Finally, Testful is compared against existing automated test frameworks (jAutoTest, randoop, etoc). TestFul is found to produce better tests than its competitors.

TestFul makes use of evolutionary algorithms to produce optimal coverage ratios. Evolutionary algorithms are based on genetics and natural selection where solutions are evaluated based on their fitness in a given context. Mutation is introduced by applying small changes to surviving solutions (individuals). Specifically, TestFul utilizes a hybrid multi-objective evolutionary algorithm, which extends the traditional evolutionary algorithm to allow individuals to improve themselves by applying a local search to find a close optimum value. TestFul reuses an object’s state to exercise different features. The article details the intricacies of this algorithm.

The creation of a test follows these basic operations:

  • Assign: Assigns a primitive value.
  • Create: Using the available constructors, create an object and store a reference to it in the context.
  • Invoke: Invoking a method. The parameters and receiving objects are extracted from the variables in the context.

The mutation of a test works by selecting an operation and doing one of the following:

  • Remove it: Removing an operation may allow the execution of a certain branch.
  • Add: Add another operation that precedes or follows the operation to alter the state of certain conditions.
  • Change the values: Change a stored context variable that the operation uses.

TestFul was found to produce higher statement and branch coverage than the others compared in the study (jAutoTest, randoop, etoc). TestFul was also found to be particularly helpful when applied to complex classes because it puts objects in useful states. In the future, TestFul will know how to better utilize information about its context to determine the interesting states of an object.