: Narayan Ramasubbu, Rajesh Krishna BalanVenue
: International Conference on Software EngineeringDate
: 2012Participant Selection
: volunteers companiesData Collection Method
: Observation, Project Artifact(s)
Software is difficult to estimate in its early stages: this is something that many companies find. This paper looks at possible causes of this, using three CMMI Level-5 firms to observe; despite this ranking, the three firms often had large estimate inaccuracies for early stages (up to about 300% variance). The companies involved experienced significant losses, considering they predominately used fixed-price contracting, where re-pricing was not an option.
Solutions were designed in three steps:
- use action research to identify the reasons for inaccurate estimates based on current methods
- develop a new cost-estimation solution that could overcome found problems
- field-test the new solution over six-months
After some study, a consensus on problems the three were facing came about:
- they didn't account for differences intrinsic to different globally distributed software projects
- their methods required metrics that weren't available at the beginning of the project
- methods were very sensitive to the experience of the manager
Researchers came up with 16 metrics to base early estimations on in place of the old system, most based off history of the teams/companies at work, such as newness of technologies and size of team.
Having developed a new early-estimation procedure, researchers began to observe it in practice at these three companies. For a six-month period, they viewed a total of 219 projects across the three companies and calculated a Magnitude of Error in Estimation (MRE). Of the 219, only 15.18% of the projects benefited greater than 25%, and 63.39% of projects benefited less than or equal to 10%. As a result, the article concludes on a bittersweet note: the researchers saw some positive change in estimation inaccuracies, but not as much as they desired.