Chapter 11. Model Assessment and Scoring

In the previous chapter we built a model. In this chapter, we are going to discuss different ways of assessing and improving the results of a model. In addition, you will also begin to learn how to use your models in the real world. Specifically, we will cover the following topics:

• Contrasting model assessment with the Evaluation phase
• Model assessment using the Analysis node
• Modifying CHAID settings
• Model comparison using the Analysis node
• Model assessment and comparison using the Evaluation node
• Scoring new data
• Exporting predictions

Contrasting model assessment with the Evaluation phase

As we briefly discussed in Chapter 1, Introduction to Data Mining and Predictive Analytics, model assessment (a modeling phase task) is quite different from the Evaluation phase. Some of the same tools can apply to both, but the stage of the project and the thought process is quite different. During model assessment, you are potentially comparing a large number of models. You may even try dozens of variations of algorithms, settings, and modifications to the data.

Therefore, you need easy, objective criteria on which to rank these models. Our colleagues and management simply will not have the time to be brought in to judge the efficacy of dozens of models so we need to narrow it down to just a couple of models before the Evaluation phase begins. Tom Khabaza, one of the original authors of CRISP-DM has written the Nine Laws of Data Mining (http://khabaza.codimension.net/index_files/9laws.htm) and the 8th Law of Data Mining is the Value...

Summary

This chapter discussed different ways of assessing the results of a model by using the Analysis and Evaluation nodes. You also began to learn how to use your models in the real world by learning how to score new data and how to export those predictions.