94 L1.02: Section 1

Section 1: Choosing a numerical indicator of how well a model fits a dataset

In an earlier topic we computed a numerical indicator of the noise of a measurement process, the standard deviation around the average measurement. If we had a similar indicator of the scatter of the data around a model, we could use it to guide our search for the best set of model parameters. Even better, we might be able to get a computer program to help in the search since computers are good at working with numbers. Our original method of judging a model, visually comparing the data and model graphs, is easy for people but beyond the current capabilities of computers.

While scatter indicators of this kind as often referred to as measuring “goodness of fit”, keep in mind that they are showing the amount of deviation, so smaller is better (these are actually badness-of-fit indicators). We will find the best model by adjusting the parameters to make the indicator as close to zero as possible.

We want to minimize both positive and negative deviations, since here we care about how close the data is to the model, but not whether it is higher or lower. In this situation, the most productive way to treat positive and negative deviations the same is to square each deviation (i.e., multiply it by itself, a process that produces positive results for both positive and negative numbers). We will then minimize some indicator based on these squared deviations.

In spreadsheets that arranged similarly to Models.xls (input data in column A, output data in column B, model prediction in column C, and data-model deviation in column D), it is natural to put these squared deviations in column E. This can be done by putting the formula “=D3^2” into cell E3, then spreading that formula down all the data rows.

Example 1: Add a squared-deviations row to the US-population spreadsheet made in an earlier topic

What number will be used to represent the squared-deviation values?

If we are going to make decisions about parameter settings by looking at a numerical goodness-of-fit indicator, that indicator must be a single number so we can decide which settings are better by choosing those that give the lowest indicator value. What indicator should be used to summarize this column of squared deviations? One obvious possibility, which turns out to be the most generally useful, is the sum of the squared deviations. This can be computed by applying the spreadsheet SUM function to column E, as by putting the formula “=SUM(E3:E12)” into some empty cell such as H8.

An alternative indicator that is useful in a few situations is simply the largest squared deviation (which comes from the largest deviation, whether positive or negative). This can be computed with the spreadsheet MAX function, using the formula “=MAX(E3:E12)” in this case. Other indicators are sometimes constructed that take into account the economic cost in a particular situation of different deviation sizes and/or directions. But the sum of the squared deviations is by far the most commonly used indicator for fitting models to data, and we will use it in this course except where another indicator is specifically asked for.

Example 2: Improve the earlier population model by using the sum-of-squared-deviations indicator.

The exponential-model parameter settings a = 2.8 and r = 0.03 that were found earlier fit the data pretty well, but of course other nearby settings of the parameters, such as would change the graph so little we could not tell by looking whether one of these models is slightly better than the other. But the numerical sum-of-squared-deviations calculation in H8 provides a way to choose among nearby model settings to get the very best fit.

[a] Here is the worksheet result from the initial model, which was chosen by setting the first parameter of the model to 2.8 (the data value at x = 0), then adjusting the growth rate until the graphs were close.

So for a model of this data, the sum of squared deviations equals about 4.91.

[b] If the initial value for the model is increased by 0.1 million (to 2.9 million), the sum of squared deviations drops from about 4.91 to 3.24, a noticeable improvement. So the model can be considered slightly better.

Further small adjustments to G3 might reduce the standard deviation further, but it would be very tedious to try all the settings needed to find the very best fit, especially since we would need to explore values for the growth rate in G4 as well. Additionally, since a change in one of the parameters will change the effect of the other parameter, several rounds of adjustment would be needed. This process would be even more difficult for models that have three or more parameters.

But tedious tasks are what computers do best. Most spreadsheets have a built-in routine that will systematically adjust parameters to a result you ask for (in this case, the minimum value for the goodness-of-fit indicator). We will use such a built-in routine, Solver, as a way to automate the model-fitting process.