101 L1.09: Section 8

Section 8: Using alternative goodness-of-fit indicators

The standard deviation (or the sum of squared deviations on which it is based) is an example of what is called a “cost” function, a computed value that an adjustment or decision process tries to make as small as possible. While standard deviation is by far the most commonly used cost function in modeling processes, there are other possible ways to measure a model’s quality that can be more appropriate in some circumstances. Usually these different cost functions are designed to reflect differences in economic costs of different errors (such as when an overestimate has more expensive consequences than an under­estimate), to limit the effect of big deviations on the fitting process (so that a single bad value will not change the model vary much), or to allow for cases where the importance of a deviation depends on the size of the deviation relative to the data value (which is often true for exponential models).

If you use an alternative goodness-of-fit indicator to find a “best-fit” model, make an explicit statement about what indicator was minimized. This is needed both so that people can decide how to use the model, and so that they would be able to reproduce your results from the same data.

Using the maximum deviation as an alternative goodness-of-fit indicator

Standard deviation is a good description of the size of the typical deviation between a model and its data. But in some situations you may be more concerned about the maximum difference, rather than about the size of the typical difference. This can happen when you want to be able to identify a deviation amount that people can confidently expect will not be exceeded (although in that case you will usually need to allow some extra margin based on the typical random noise).

If you want to minimize the greatest difference between the population model and any individual population value, you would use the indicator “=MAX(E3:E12)” (placed in a spare cell such as H9) rather than “=SUM(E3:E12)”. If Solver is asked to minimize a cell with this formula (e.g., H9), it will adjust G3 and G4 (and any other parameters) in ways that may make the standard deviation higher, but will make the maximum distance above or below the data as small as is possible for the chosen kind of model.

Example 11: Find the exponential model that minimizes the maximum deviation from the census data.

Note that the best-fit parameters using this indicator are slightly different from those that arose from minimizing the sum of squared deviations: a is now 3.42 rather than 3.11, and r is 2.79% instead of 2.89%. This would not make much visible difference in the graph (since the slightly-higher starting value is mostly balanced by the slightly-lower growth rate), but the maximum squared deviation has been cut about in half, to 0.43 rather than 0.95. On the other hand, the sum of squared deviations is more than twice as large (3.59 rather than 1.72), resulting in a standard deviation that is 44% larger. Such trade-offs are typical with different cost functions—the model will be better for what you focus on, but not quite as good for other purposes.

Relative standard deviation

A relative deviation is the data-minus-model difference divided by the value of the model. This ratio can be expressed as a decimal fraction or a percentage – a few percent would be typical of most situations. Relative deviations can be useful when the model will not have zero values (in which case the division would not work) and there is a natural zero to the data scale (so that the ratio has corresponds to some real relationship).

If we wish to use relative deviations in a worksheet like Models.xls, we replace the formula “=B3-C3” in cell D3 (and similarly other column-D cells) by the formula “=(B3-C3)/C3”. Nothing else in the worksheet need be changed. Now Solver will minimize the sum of squared relative deviations, since that is now what is being computed in H8. The relative standard deviation can be computed from that sum in same way as before, with the formula “=SQRT(H8/(10-2))” in this case, which has 10 data points and 2 parameters.

The worksheet for the above example should look similar to this:

The reason that these model parameters are slightly different from those that Solver found when minimizing the absolute standard deviation is that using relative standard deviation means that deviations from the smaller values count more in the fitting process than they did before. This is because a deviation of 0.1 million is about 3.5% of the 2.8 million for 1780, while that same 0.1 deviation is only 0.25% of the 39.8 million population 90 years later.

Exponential models are good possible uses for relative standard deviation since such models are based on constant-percentage relative growth or decay, and never have a zero value that would make it impossible to compute the ratio. However, absolute standard deviation is still the most usual fitting method even for this kind of model, and should be used unless you have a specific reason to use some other cost function.