Working with ridge regression

Ridge regression, also referred to as L2 regularization, is a commonly used technique to alleviate overfitting in linear regression models by penalizing the magnitude of the estimated coefficients in the resulting model.

Recall that in an SLR model, we seek to minimize the sum of the squared differences between our predicted and actual values, which we refer to as the least squares method. The loss function we wish to minimize is the RSS:

RSS = ∑ i=1 n (y i − (β 0 + ∑ j=1 p β j x ij)) 2

Here, y i is the actual target value, β 0 is the intercept term, {β j} are the coefficient estimates for each predictor, x ij, and the summations are overall observations and predictors.

Purely minimizing the RSS would give us an overfitting model, as represented by the high magnitude of the resulting coefficients. As a remedy, we could apply...