Stan Pickens, Strasz Project Manager
September 12, 2014
Every knowledge worker is familiar with providing estimates, one of the business world’s necessary evils. Even in the best of circumstances, a well-considered estimate can be little more than an educated guess. But did you know that there’s a method to calculate the accuracy of estimates? This blog post explains how to do it.
Every knowledge worker is familiar with providing estimates, one of the business world’s necessary evils. Even in the best of circumstances, a well-considered estimate can be little more than an educated guess, and it doesn’t take long to become uncomfortable about the whole affair.
Despite the frequency of making estimates, few workers track the accuracy of their estimates over time. This is especially surprising, given that repeated estimates are made about the same basic processes. In the assessment industry, for example, we repeatedly make estimates about how long it takes to write a new item, assemble a test, etc.
One likely reason for not tracking the accuracy of estimates is the fear of diving into the black pool of statistics. For the typical office worker, statistics can create a sense of dread on par with global warming and killer bees. But instead of using statistics, there are simpler methods to measure the accuracy of estimates. One such method, which calculates R² (pronounced “R-squared”), is described in [Lo and Gao 1997]¹.
R² ranges from 0.0 to 1.0, where an R² of 0.0 represents the poorest accuracy and 1.0 represents perfect accuracy. We can calculate R² by using a simple spreadsheet, shown in the figure below. The sample spreadsheet calculates R² for estimates given on 9 days (Column A).
The three preliminary steps to computing R² are the following:
- Calculate the average of the actuals (Cell C11, or 5.11).
- Calculate the sum of all [(actual) – (estimate)]² (CELL E11, or 27)
- Calculate the sum of all [(actual) – (average of the actuals)]² (CELL G11, or 92.89)
With the calculated values in hand, R² is calculated using the formula given in cell G13, the result of which is 0.709. An R² of 0.709 for your estimates is not bad (pretty good, in fact!).
Naturally, the math-minded readers need to see the actual formula for R²:
Note that the denominator can become zero when all of the actuals have the same value, which would cause a divide by zero check in a spreadsheet. Also, the numerator can become very large when the actuals grossly exceed the estimates, which would then cause R² to become negative. The calculation of R² is invalid if divide by zero occurs or the result is negative.
CITATIONS:
1. Lo, B., and Gao, X. (2007). Assessing software cost estimation models: criteria for accuracy, consistency and regression. Australasian Journal of Information Systems, 5(1).
Stan Pickens, Strasz Project Manager
September 12, 2014
Every knowledge worker is familiar with providing estimates, one of the business world’s necessary evils. Even in the best of circumstances, a well-considered estimate can be little more than an educated guess. But did you know that there’s a method to calculate the accuracy of estimates? This blog post explains how to do it.
Every knowledge worker is familiar with providing estimates, one of the business world’s necessary evils. Even in the best of circumstances, a well-considered estimate can be little more than an educated guess, and it doesn’t take long to become uncomfortable about the whole affair.
Despite the frequency of making estimates, few workers track the accuracy of their estimates over time. This is especially surprising, given that repeated estimates are made about the same basic processes. In the assessment industry, for example, we repeatedly make estimates about how long it takes to write a new item, assemble a test, etc.
One likely reason for not tracking the accuracy of estimates is the fear of diving into the black pool of statistics. For the typical office worker, statistics can create a sense of dread on par with global warming and killer bees. But instead of using statistics, there are simpler methods to measure the accuracy of estimates. One such method, which calculates R² (pronounced “R-squared”), is described in [Lo and Gao 1997]¹.
R² ranges from 0.0 to 1.0, where an R² of 0.0 represents the poorest accuracy and 1.0 represents perfect accuracy. We can calculate R² by using a simple spreadsheet, shown in the figure below. The sample spreadsheet calculates R² for estimates given on 9 days (Column A).
The three preliminary steps to computing R² are the following:
- Calculate the average of the actuals (Cell C11, or 5.11).
- Calculate the sum of all [(actual) – (estimate)]² (CELL E11, or 27)
- Calculate the sum of all [(actual) – (average of the actuals)]² (CELL G11, or 92.89)
With the calculated values in hand, R² is calculated using the formula given in cell G13, the result of which is 0.709. An R² of 0.709 for your estimates is not bad (pretty good, in fact!).
Naturally, the math-minded readers need to see the actual formula for R²:
Note that the denominator can become zero when all of the actuals have the same value, which would cause a divide by zero check in a spreadsheet. Also, the numerator can become very large when the actuals grossly exceed the estimates, which would then cause R² to become negative. The calculation of R² is invalid if divide by zero occurs or the result is negative.
CITATIONS:
1. Lo, B., and Gao, X. (2007). Assessing software cost estimation models: criteria for accuracy, consistency and regression. Australasian Journal of Information Systems, 5(1).