Let’s embark on a journey into the world of regularization in machine learning. I hope you will learn a lot and find as much joy in reading this book as I did in writing it.

Regularization is important for any individual willing to deploy robust machine learning (ML) models.

This chapter will introduce some context and key concepts about regularization before diving deeper into it in the next chapters. At this point, you may have many questions about this book and about regularization in general. What is regularization? Why do we need regularization for production-grade ML models? How do we diagnose the need for regularization? What are the limits of regularization? What are the approaches to regularization?

All the foundational knowledge about regularization will be provided in this chapter in the hope of answering all these questions. Not only will this give you a high-level understanding of what regularization is but it will also allow...

In this chapter, you will have the opportunity to generate a toy dataset, display it, and train basic linear regression on that data. Therefore, the following Python libraries will be required:

• NumPy
• Matplotlib
• scikit-learn

“Regularization in ML is a technique used to improve the generalization performance of a model by adding additional constraints to the model’s parameters. This forces the model to use simpler representations and helps reduce the risk of overfitting.

Regularization can also help improve the performance of a model on unseen data by encouraging the model to learn more relevant, generalizable features.”

This definition of regularization, arguably good enough, was actually generated by the famous GPT-3 model when given the following prompt: Detailed definition of regularization in machine learning. Even more astonishing, this definition passed several plagiarism tests, meaning it’s actually fully original text. Do not worry if you do not yet understand all the words in this definition from GPT-3; it is not meant for beginners. But you will fully understand it by the end of this chapter.

Note

GPT-3, short for Generative Pre...

Having gained some intuition regarding what constitutes a suitable fit, as well as understanding examples of underfitting and overfitting, let us now delve into a more precise definition and explore key concepts that enable us to better comprehend regularization.

Bias and variance

Bias and variance are two key concepts when talking about regularization. We can define two main kinds of errors a model can have:

• Bias is how bad a model is at capturing the general behavior of the data
• Variance is how bad a model is at being robust to small input data fluctuations

Those two concepts, in general, are not mutually exclusive. If we take a step back from ML, there is a very common figure to visualize bias and variance, assuming the model’s goal is to hit the center of a target:

Figure 1.7 – Visualization of bias and variance

Let’s describe those four cases:

• High bias and low variance...

Having the right diagnosis for a model is crucial, as it allows us to choose the strategy more carefully to improve the model. But from any diagnosis, many paths are possible to improve the model. Those paths can be separated into three main categories, as proposed in the following figure:

Figure 1.17 – A proposed categorization of regularization types: data, model architecture, and model training

At the data level, we may have the following tools for regularization:

• Adding more data, either synthetic or real
• Adding more features
• Feature engineering
• Data preprocessing

Indeed, the data is of extreme importance in ML in general, and regularization is no exception. We will see many examples throughout the book of regularizing data.

At the model level, the following methods may be used for regularization:

• Choosing a more or less simple architecture
• In deep...

We started this chapter by demonstrating, with several real-world examples, that regularization is the key to success in ML in a production environment. Along with several other methods and best practices, a robustly regularized model is necessary for production. In production, unseen data and edge cases will appear on a regular basis, thus any deployed model must have an acceptable response to such cases.

We then walked through some key concepts of regularization. Overfitting and underfitting are two common problems in ML and relate somehow to bias and variance. Indeed, an overfitting model has high variance, while an underfitting model has high bias. Thus, to perform well, a model is required to have low bias and low variance. We explained how, no matter how good a model can get, unavoidable bias limits its performance. Those key concepts allowed us to propose a method to diagnose bias and variance using the performance of both the training and validation sets, as well...