Chapter 4. Data Quality and Exploration

The previous chapter introduced the general data structure that is used in Modeler. You learned how to read and display data, and you were introduced to the concepts of the measurement level and the field roles. Now that you know how to bring data into Modeler, the next step is to assess the quality of the data. In this chapter you will:

• Get an overview of the Data Audit node options
• Go over the results of the Data Audit node
• Be introduced to missing data
• Discuss ways to address missing data

Once your data is in Modeler, you are ready to start exploring and become familiar with the characteristics of the data. You should review the distribution of each field so that you can become familiar with a dataset, but also so that you can identify potential problems that may arise. For continuous fields, you will want to inspect the range of values. For categorical fields, you will want to take a look at the number of distinct values. You will also have to consider...

Data Audit node options

When data is first read into Modeler, it is important to check the data to make sure it was read correctly. Typically, using a Table node can help you get a sense of the data and inform you of some potential issues that you may have. However, the Data Audit node is a better alternative to using a Table node, as it provides a more thorough look at the data.

Before modeling takes place, it is important to see how records are distributed within the fields in the dataset. Knowing this information can identify values that, on the surface, appear to be valid, but when compared to the rest of the data are either out of range or inappropriate. Let's begin by opening a stream that has the modifications we made in the previous chapter:

1. Open the Data Quality and Exploration stream.

Note

This simple stream contains the Demographic data file that has been linked to the Var. File source, along with the modifications we previously made in the Types tab. In order for this stream to function...

Summary

This chapter focused on understanding your data. You learned about the different options available in the Data Audit node. You also learned how to look over the results of the Data Audit node to get a better feel for your data and to identify potential problems your data may have. Finally, you were introduced to the topic of missing data, and several ways to address this issue were discussed.

In the next chapter, we will begin to fix some of the problems that we found in the data. Specifically, we will use the Select node to choose the appropriate sample for our analysis. We will also use the Reclassify node to modify fields so that the distributions are appropriate for modeling. We will also use other nodes to rectify other concerns.