This chapter discussed why model evaluation is important in supervised machine learning and looked at several important metrics that are used to evaluate regression and classification tasks. We saw that while regression models were fairly straightforward to evaluate, the performance of classification models could be measured in a number of ways, depending on what we want the model to prioritize. Besides numerical metrics, we also looked at how to plot precision-recall and ROC curves to better interpret and evaluate model performance.
After this, we talked about why evaluating a model by calculating the prediction error on the data that the model was trained on was a bad idea, and how testing a model on data that it has already seen would lead to the model having a high variance. With this, we introduced the concept of having a hold-out dataset and why K-fold cross-validation is a useful strategy to have, along with sampling techniques that ensure that the model training and evaluation...
