An Application of Six Sigma and Simulation in Software Testing Risk Assessment

Author(s): Vojo Bubevski
Venue: 2010 Third International Conference on Software Testing, Verification and Validation
Date: 2010

Type of Experiement: Controlled Experiment


This paper shows how a Monte Carlo Simulation can be used in a Six Sigma DMAIC(Define, Measure, Analyze, Improve, Control) approach to Software Testing. Raw data used by this experiment was from 41 weeks of testing on the Galileo spacecraft Control and Data System. Simulations were created using the raw data and then compared with the actual data.
To test the new method a pretend project was made which was supposed to be completed in 32 weeks. The pretend project was analyzed at the end of the 23rd week of testing with 10 weeks of testing remaining.
Running a Monte Carlo Simulation with data from the first 23 weeks produced results which indicated that the product would not be able to meet its due date with the desired level of quality. In order to quantify an improvement another simulation was created for the next 18 weeks. This new simulation showed that there would be 13 defects from Weeks 33 - 41. This means that in order to meet the quality goals either more resources should be allocated to the project or the deadline for the project should be moved to week 41. If no further resources were allocated, then by Week 41 there was a 74% probability that there would be zero defects reported.
The paper concluded that using Monte Carlo Simulations in a Six Sigma DMAIC is better than using static analysis methods. When the simulated data was compared to the actual data it was found to be accurate within 7.14%. This new approach helps to detect failures to meet quality assurance goals while assigning quality confidence levels to scheduled product releases. Finally, the techniques used in the experiment are shown to comply with CMMI® Level 4 and CMMI® Level 5.