Chapter 6. Model Evaluation

Note

Learning Objectives

By the end of this chapter, you will be able to:

• Explain model evaluation, accuracy, null accuracy, and the limitations of accuracy

• Explain imbalanced datasets and confusion matrices

• Evaluate sensitivity, specificity, precision, FPR, ROC curves, and AUC scores

• Evaluate the classification threshold

Note

In this chapter, we will learn how to evaluate a model using accuracy. We will evaluate the model with sensitivity, specificity, precision, FPR, ROC curves, and AUC curves. Lastly, we will apply a classification threshold on the model.

Introduction

In this chapter, we will learn about some different evaluation techniques other than accuracy. For any data scientist, the first step after building a model is to evaluate it, and the easiest way to evaluate a model is through its accuracy. However, in real-world scenarios, we often deal with datasets where accuracy is not the best evaluation technique. This chapter explores core concepts such as imbalanced datasets and how different evaluation techniques can be used to work through these imbalanced datasets. The chapter begins with an introduction to accuracy and its limitations. It then explores the concepts of null accuracy, imbalanced datasets, sensitivity, specificity, precision, false positives, ROC curves, and AUC scores.

Accuracy

To understand accuracy properly, first let's explore model evaluation. Model evaluation is an integral part of the model development process. Once you build your model and execute it, the next step is to evaluate your model. A model is built on a training dataset, and evaluating a model's performance on the same training dataset is a bad practice in data science. Once a model is trained on a training dataset, it should be evaluated on a dataset that is completely different from the training dataset. This dataset is known as the test dataset. The objective should always be to build a model that generalizes, which means the model should produce similar (but not the same) results, or relatively similar results, on any dataset. This can only be achieved if we evaluate the model on data that is unknown to it.

The model evaluation process requires a metric that can quantify a model's performance. The simplest metric for model evaluation is accuracy. Accuracy is the fraction of predictions...

Imbalanced Datasets

Imbalanced datasets are a distinct case for classification problems where the class distribution varies between the classes. In such datasets, one class is overwhelmingly dominant. In other words, the null accuracy of an imbalanced dataset is very high. Consider an example of credit card fraud. If we have a dataset of credit card transactions, then we will find that, of all the transactions, a very miniscule number of transactions were fraudulent and the majority of transactions were normal transactions. If 1 represents a fraudulent transaction and 0 represents a normal transaction, then there will be many 0s and hardly any 1s. The null accuracy of the dataset may be more than 99%. This means the majority class (in this case, 0) is overwhelmingly greater than the minority class (in this case, 1). Such sets are imbalanced datasets. The following figure shows a generalized scatter plot of an imbalanced dataset, where the stars represent the minority class and the circles...

Confusion Matrix

A confusion matrix describes the performance of the classification model. In other words, confusion matrix is a way to summarize classifier performance. The following figure shows a basic representation of a confusion matrix:

Figure 6.5: Basic representation of a confusion matrix

The following code is an example of a confusion matrix:

from sklearn.metrics import confusion_matrix
cm=confusion_matrix(y_test,y_pred_class)
print(cm)

The following figure shows the output of the preceding code:

Figure 6.6: Example confusion matrix

These are the meanings of the abbreviations used in the preceding figure:

• TN (True negative): This is the count of outcomes that were originally negative and were predicted negative.

• FP (False positive): This is the count of outcomes that were originally negative but were predicted positive. This error is also called a type 1 error

• FN (False negative): This is the count of outcomes that were originally positive but were predicted negative. This error is also...

Summary

In this chapter, we learned about model evaluation and accuracy. We learned how accuracy is not the most appropriate technique for evaluation when our dataset is imbalanced. We also learned how to compute a confusion matrix using scikit-learn and how to derive other metrics, such as sensitivity, specificity, precision, and false positive rate. Finally, we learned how to use threshold values to adjust metrics and how ROC curves and AUC scores help us evaluate our models. It is very common to deal with imbalanced datasets in real-life problems. Problems such as credit card fraud detection, disease prediction, and spam email detection all have imbalanced data in different proportions.