98 L1.06: Section 5

Section 5: Identifying and Removing Outliers in Data

Some datasets include a few points that do not fit into the overall pattern. These points are called “outliers”, and require special handling if a model is to be fit to the dataset. In general, the approach to modeling data with outliers is to exclude them from the fitting process but report them separately along with the model formula.

Sometimes outliers are simply mistakes in the data. Measurement is not a perfect process, and errors in a measurement device or in writing down the measurement results can lead to bad values in a dataset. One benefit of looking at the overall pattern of the data is that it usually will reveal any substantial errors of this kind.

On the other hand, some outliers are accurate measurements but report anomalies, situations that are different from the typical situation in which the other measurements were made. Measurements of Sunday pedestrian traffic in downtown Austin, for example, would show an outlier each spring due to the Capital 10K race.

It is important to report outliers, so that people depending on measurements similar to those that produced the data are alerted to the possibility of large deviations from the general trend of the data. Even if the outlier is a mistake, it is an indication that users should watch out for similar mistakes. If the outlier is an anomaly, it is possible that it is the most important part of the data. For example, we would want the people who design bridges to design for the maximum load and not the typical load.

When using Models.xls to find best-fit models, we can exclude outliers from the fitting process by erasing the content of the column E cell for each row that contains outlier data. This means that the deviation associated with the outlier value is not counted in computing the standard deviation, and thus does not influence the fitting process.

Example 7: Using the dataset shown to the right:

1. Fit and report on a linear model using all of the data
2. Fit and report on a linear model with the outlier excluded.

Solution approach:

1. Copy the data to a Linear Model worksheet in Models.xls.
2. Spread columns C, D, and E down to match the data, as usual.
3. Make a graph of the data and model together, as usual.
4. Use Solver to find the best-fit parameters and standard deviation.
5. Erase cell E6, since the graph shows the data in row 6 is an outlier.
6. Use Solver again, to find results with the outlier excluded.

Notice that removal of the outlier makes a great difference in the size of the standard deviation, since the way standard deviations are computed emphasizes any large deviations.

Is this particular outlier a mistake or an anomaly? You can’t tell from the numbers, since any type of outlier consists of big deviations. But in this case the caption for the data indicates a process that logically must change smoothly. Thus this outlier is a mistake. Examination of the data suggests that the “35.3” y value for the outlier should have been about 100 higher, so perhaps an actual measurement of “135.3” was copied incorrectly.

Example 8: For the U.S. airline-passenger data provided to the right below:

1. Report the best-fit linear model formula and its standard deviation, using all the data points provided.
2. Examine the graph and list of deviations in column D to identify any data points which are outliers from the data trend from 1990 to 2000.
3. Report the best-fit line and σ when the outliers are excluded.
4. In this case, are the outlier points mistakes or anomalies?
5. How many 2006 passengers would there have been (to the nearest million) if US air traffic had continued its 1990-2000 trend?
6. Has the airline industry fully recovered its 1990-2000 trend? Why?

Worksheet showing the result of fitting the data: