Dealing with Missing Data and Outliers

The earlier chapters showed you how to read common file formats and define Variable Properties. In any project, as you pull together the data that helps you address your business question or research question, you must spend some time gaining an understanding of your data via a data audit. Simple procedures such as Frequencies, Descriptives, or Examine can give you a summary understanding of each variable via statistical and graphical means. In addition, the data audit should focus on unusual/extreme values and the nature and extent of missing data.

The topics covered in this chapter include the following:

Outliers:

• Frequencies for a histogram and percentile values
• Descriptives for standardized scores
• The Examine procedure for extreme values and boxplot
• Detecting multivariate outliers using the Regression procedure

Missing data:

• Missing...

Outliers

An outlier is an observation that lies an unusual distance from other observations. There is a judgmental element in deciding what is considered unusual, and it helps to work with the subject-matter expert in deciding this. In exploratory data analysis, there are two activities that are linked:

• Examining the overall shape of the graphed data for important features
• Examining the data for unusual observations that are far from the mass or general trend of the data

Outliers are data points that deserve a closer look. The values could be real data values accurately recorded or the values could be misrecorded or otherwise flawed data. You need to discern what is the case in your situation and decide what action to take.

In this section, we consider statistical and graphical ways of summarizing the distribution of a variable and detecting unusual/extreme values. IBM SPSS...

Missing data

Just as you ought to assess outliers and extreme values in the variables being analyzed, you should also assess the missing responses in the variables being analyzed. For a given variable, what number or fraction of responses is missing? What is or are the mechanisms by which missing values happen? Is the missingness in a variable related to values on another variable or perhaps that same variable? Fully addressing these questions in the context of your data can be hard work, and a full discussion is beyond the scope of this book. Here, we briefly address why missing data matters and show some analyses that you can do.

Why should you be concerned about missing data?

There are two reasons:

• Statistical efficiency
• Bias

Statistical efficiency has to do with the relationship between sample size and precision. If your data is a random sample from a population, then along...

Summary

In the early stages of working with a dataset, you gain data understanding by at least selectively performing outlier analysis and missing value analysis. IBM SPSS Statistics offers many useful facilities for outlier analysis. In this chapter, we looked at ways of generating histograms, percentiles, z-scores, and boxplots to gain an understanding of outliers. In addition, most procedures in IBM SPSS Statistics produce a simple summary table of valid and missing cases. We also saw how to look at missing value patterns and perform mean substitution. In the next chapter, we turn to visually exploring the data through charts.