Web engineering - Measuring Effort Prediction Power and Accuracy

Web engineering - Measuring Effort Prediction Power and Accuracy

• 1.
  Measuring Effort PredictionPower and Accuracy By Nosheen Qamar University of Lahore, Pakistan
• 2.
   An effortestimation model m uses historical data of finished projects to predict the effort of a new project. Some believe this is enough to provide accurate effort estimates. However, to gauge the accuracy of this model we need to measure its predictive accuracy.  To measure a model’s predictive accuracy first calculate the predictive power for each of a set of new projects p1 to pn that used the effort estimation model m. Once predictive power for p1 to pn has been obtained, their values are aggregated, which gives the predictive power of model m and hence its corresponding predictive accuracy.
• 3.
   The mostcommon approaches to date for measuring predictive power of effort estimation models are:  The Mean Magnitude of Relative Error (MMRE)  The Median Magnitude of Relative Error (MdMRE)  The Prediction at level n (Pred(n))
• 4.
   The basisfor calculating MMRE and MdMRE is to use the Magnitude of Relative Error (MRE), defined as:  where e is the actual effort and ê is the estimated effort.  The mean of all MREs is the MMRE, calculated as:
• 5.
   As themean is calculated by taking into account the value of every estimated and actual effort from the data set employed, the result may give a biased assessment of a model’s predictive power when there are several projects with large MREs.  An alternative to the mean is the median, which also represents a measure of central tendency, as it is less sensitive to the existence of several large MREs. The median of MRE values for the number i of observations (data values) is called the MdMRE.  MMRE, MdMRE are taken as the de facto standard evaluation criteria to measure the predictive power of effort estimation models
• 6.
   In orderto calculate the predictive accuracy of a given effort estimation model m, based on a given data set of finished projects d, we do the following: ◦ Divide the data set d into a training set t and a validation set v. It is common to create training sets that use 66% of the projects from the complete data set, leaving 34% for the validation set. ◦ Using t, produce an effort estimation model m (if applicable). ◦ Using m, predict the effort for each of the projects in v, simulating new projects for which effort is unknown.  Once done, we will have, for each project in v, an estimated effort ê, calculated using the model m, and also the actual effort e that the project actually used. We are now able to calculate the predictive power (MRE) for each project in the validation set v. The final step, once we have obtained the predictive power for each project, is to aggregate these values to obtain MMRE, MdMRE for v, which is taken to be the same for m.  Calculated MMREs and MdMREs with values up to 0.25 indicate good prediction models.
• 7.
   We introducednumerous techniques for obtaining effort estimates for a new project, and all have been used, each with a varying degree of success. Therefore the question that is often asked is: Which of the techniques provides the most accurate prediction?  To date, the answer to this question has been simply “it depends”. Algorithmic models have some advantages over machine learning and expert opinion, such as: ◦ Allowing users to see how a model derives its conclusions, an important factor for verification as well as theory building and understanding of the process being modelled. ◦ The need to be specialised relative to the local environment in which they are used.
• 8.
   Despite theseadvantages, no convergence on which effort estimation technique has the best predictive power has yet been reached, even though comparative studies have been carried out over the last 15 years.  One justification is that these studies have used data sets with differing characteristics (e.g. number of outliers,1 amount of collinearity,2 number of variables, number of projects) and different comparative designs.  Shepperd and Kadoda [35 ]presented evidence for the relationship between the success of a particular technique and training set size, nature of the “effort estimation” function, and characteristics of the data set. They concluded that the “best” prediction technique that can work on any type of data set may be impossible to obtain