7. Model Evaluation

Overview

This chapter is an introduction to how you can improve a model's performance by using hyperparameters and model evaluation metrics. You will see how to evaluate regression and classification models using a number of metrics and learn how to choose a suitable metric for evaluating and tuning a model.

By the end of this chapter, you will be able to implement various sampling techniques and perform hyperparameter tuning to find the best model. You will also be well equipped to calculate feature importance for model evaluation.

Introduction

In the previous chapters, we discussed the two types of supervised learning problems, regression and classification, followed by ensemble models, which are built from a combination of base models. We built several models and discussed how and why they work. However, that is not enough to take a model to production. Model development is an iterative process, and the model training step is followed by validation and updating steps, as shown in the following figure:

Figure 7.1: Machine learning model development process

This chapter will explain the peripheral steps in the process shown in the preceding flowchart; we will discuss how to select the appropriate hyperparameters and how to perform model validation using the appropriate error metrics. Improving a model's performance happens by iteratively performing these two tasks. But why is it important to evaluate your model? Say you've trained your model and provided some hyperparameters...

Importing the Modules and Preparing Our Dataset

In the previous exercises and activities, we used terms such as Mean Absolute Error (MAE) and accuracy. In machine learning terms, these are called evaluation metrics and, in the next sections, we will discuss some useful evaluation metrics, what they are, and how and when to use them.

Note

Although this section is not positioned as an exercise, we encourage you to follow through this section carefully by executing the presented code. We will be using the code presented here in the upcoming exercises.

We will now load the data and models that we trained as part of Chapter 6, Ensemble Modeling. We will use the stacked linear regression model from Activity 6.01, Stacking with Standalone and Ensemble Algorithms, and the random forest classification model to predict the survival of passengers from Exercise 6.06, Building the Ensemble Model Using Random Forest.

First, we need to import the relevant libraries:

import pandas as...

Evaluation Metrics

Evaluating a machine learning model is an essential part of any project: once we have allowed our model to learn from the training data, the next step is to measure the performance of the model. We need to find a metric that can not only tell us how accurate the predictions made by the model are, but also allow us to compare the performance of a number of models so that we can select the one best suited for our use case.

Defining a metric is usually one of the first things we should do when defining our problem statement and before we begin the exploratory data analysis, since it's a good idea to plan ahead and think about how we intend to evaluate the performance of any model we build and how to judge whether it is performing optimally. Eventually, calculating the performance evaluation metric will fit into the machine learning pipeline.

Needless to say, evaluation metrics will be different for regression tasks and classification tasks, since the output...

Splitting a Dataset

A common mistake made when determining how well a model is performing is to calculate the prediction error on the data that the model was trained on and conclude that a model performs really well on the basis of a high prediction accuracy on the training dataset.

This means that we are trying to test the model on data that the model has already seen, that is, the model has already learned the behavior of the training data because it was exposed to it—if asked to predict the behavior of the training data again, it would undoubtedly perform well. And the better the performance on the training data, the higher the chances that the model knows the data too well, so much so that it has even learned the noise and behavior of outliers in the data.

Now, high training accuracy results in a model having high variance, as we saw in the previous chapter. In order to get an unbiased estimate of the model's performance, we need to find its prediction accuracy...

Performance Improvement Tactics

Performance improvement for supervised machine learning models is an iterative process, and a continuous cycle of updating and evaluation is usually required to get the perfect model. While the previous sections in this chapter dealt with the evaluation strategies, this section will talk about model updating: we will discuss some ways we can determine what our model needs to give it that performance boost, and how to effect that change in our model.

Variation in Train and Test Errors

In the previous chapter, we introduced the concepts of underfitting and overfitting, and mentioned a few ways to overcome them, later introducing ensemble models. But we didn't talk about how to identify whether our model was underfitting or overfitting to the training data.

It's usually useful to look at the learning and validation curves.

Learning Curve

The learning curve shows the variation in the training and validation errors with the training...

Summary

This chapter discussed why model evaluation is important in supervised machine learning and looked at several important metrics that are used to evaluate regression and classification tasks. We saw that while regression models were fairly straightforward to evaluate, the performance of classification models could be measured in a number of ways, depending on what we want the model to prioritize. Besides numerical metrics, we also looked at how to plot precision-recall and ROC curves to better interpret and evaluate model performance. After this, we talked about why evaluating a model by calculating the prediction error in relation to the data that the model was trained on was a bad idea, and how testing a model on data that it has already seen would lead to the model having a high variance. With this, we introduced the concept of having a hold-out dataset and demonstrated why k-fold cross-validation is a useful strategy to have, along with sampling techniques that ensure that...