Regularization

As with any machine learning algorithm, neural networks can face the problem of overfitting when they learn patterns that are only relevant to the training set. In such a case, the model will not be able to generalize the unseen data.

Luckily, there are multiple techniques that can help reduce the risk of overfitting:

• L1 regularization, which adds a penalty parameter (absolute value of the weights) to the loss function
• L2 regularization, which adds a penalty parameter (squared value of the weights) to the loss function
• Early stopping, which stops the training if the error for the validation set increases while the error decreases for the training set
• Dropout, which will randomly remove some neurons during training

All these techniques can be added at each layer of a neural network we create. We will be looking at this in the next exercise.

Exercise 6.04: Predicting Boston House Prices with Regularization

In this exercise, you will...