Measuring Performance and Selecting Models

This chapter describes the bias and variance effects and their pathological cases, which usually appear when training machine learning (ML) models. For example, the high variance effect, also known as overfitting, is a phenomenon in ML where the constructed model explains the examples from the training set but works relatively poorly on the examples that did not participate in the training process. This occurs because while training a model, random patterns will start appearing that are normally absent from the general population. The opposite of overfitting is known as underfitting. This happens when the trained model becomes unable to predict patterns in new data or even in the training data. Such an effect can be the result of a limited training dataset or weak model design.

In this chapter, we will learn how to deal with overfitting...

Technical requirements

For this chapter, you will need the following:

• A modern C++ compiler with C++17 support
• CMake build system version >= 3.8
• Dlib library
• Shogun-toolbox library
• Shark-ML library
• Plotcpp library

The code files for this chapter can be found at the following GitHub repo: https://github.com/PacktPublishing/Hands-On-Machine-Learning-with-CPP/tree/master/Chapter03

Performance metrics for ML models

When we develop or implement a particular ML algorithm, we need to estimate how well it works. In other words, we need to estimate how well it solves our task. Usually, we use some numeric metrics for algorithm performance estimation. An example of such a metric could be a value of mean squared error that's been calculated for target and predicted values. We can use this value to estimate how distant our predictions are from the target values we used for training. Another use case for performance metrics is their use as objective functions in optimization processes. Some performance metrics are used for manual observations, though others can be used for optimization purposes too.

Performance metrics are different for each of the ML algorithms types. In Chapter 1, Introduction to Machine Learning with C++, we discussed that two main categories...

Understanding the bias and variance characteristics

The bias and variance characteristics are used to predict model behavior. They are universal keywords. Before we go any further and describe what they mean, we should consider validation. Validation is a technique that's used to test model performance. It estimates how well the model makes predictions on new data. New data is data that we did not use for the training process. To perform validation, we usually divide our initial dataset in two or three parts. One part should contain most of the data and will be used for training, while other ones will be used to validate and test the model. Usually, validation is performed for iterative algorithms after one training cycle (often called an epoch). Alternatively, we perform testing after the overall training process.

The validation and testing operations evaluate the model...

Model selection with the grid search technique

It is necessary to have a set of proper hyperparameter values to create a good ML model. The reason for this is that having random values leads to controversial results and behaviors that are not expected by the practitioner. There are several approaches we can follow to choose the best set of hyperparameter values. We can try to use hyperparameters from the algorithms we have already trained that are similar to our task. We can also try to find some heuristics and tune them manually. However, this task can be automated. The grid search technique is the automated approach for searching for the best hyperparameter values. It uses the cross-validation technique for model performance estimation.

Cross-validation

...

Summary

In this chapter, we discussed how to estimate the ML model's performance and what metrics can be used for such estimation. We considered different metrics for regression and classification tasks and what characteristics they have. We have also seen how performance metrics can be used to determine the model's behavior, and also looked at the bias and variance characteristics. We looked at some high bias (underfitting) and high variance (overfitting) problems and considered how to solve them. We also learned about the regularization approaches, which are often used to deal with overfitting. We then studied what validation is and how it is used in the cross-validation technique. We saw that the cross-validation technique allows us to estimate model performance while training limited data. In the last section, we combined an evaluation metric and cross-validation...

Further reading

• Choosing the Right Metric for Evaluating Machine Learning Models—Part 1: https://medium.com/usf-msds/choosing-the-right-metric-for-machine-learning-models-part-1-a99d7d7414e4
• Understand Regression Performance Metrics: https://becominghuman.ai/understand-regression-performance-metrics-bdb0e7fcc1b3
  
• Classification Performance Metrics: https://nlpforhackers.io/classification-performance-metrics/

• REGULARIZATION: An important concept in Machine Learning: https://towardsdatascience.com/regularization-an-important-concept-in-machine-learning-5891628907ea
• An Overview of Regularization Techniques in Deep Learning (with Python code): https://www.analyticsvidhya.com/blog/2018/04/fundamentals-deep-learning-regularization-techniques
• Understanding the Bias-Variance Tradeoff: https://towardsdatascience.com/understanding-the-bias-variance-tradeoff-165e6942b229
• Deep Learning...