Simple Linear Regression

In previous chapters, we worked with distributions of single variables. Now we will discuss the relationships between variables. In this chapter and the next chapter, we will investigate the relationship between two or more variables using linear regression. In this chapter, we will discuss simple linear regression within the framework of Ordinary Least Squares (OLS) regression. Simple linear regression is a very useful tool for estimating continuous values from two linearly related variables. We will provide an overview of the intuitions and calculations behind regression errors. Next, we will provide an overview of the pertinent assumptions of linear regression. After that, we will analyze the output summary of OLS in statsmodels, and finally, we will address the scenarios of serial correlation and model validation. As highlighted, our main topics in this chapter follow this framework:

• Simple linear regression using OLS
• Coefficients of correlation...

Simple linear regression using OLS

We will study one of the simplest machine learning models – simple linear regression. We will provide its overview within the context of OLS, where the objective is to minimize the sum of the square of errors. It is a straightforward concept related to a dependent variable (quantitative response) y and its independent variable x, where their relationship can be drawn as a straight line, approximately. Mathematically, a simple linear regression model can be written in the following form:

y = β 0 + β 1 x + ϵ

Here, β 0 is the intercept term and β 1 is the slope of the linear model. The error term is denoted as ϵ in the preceding linear model. We can see that in an ideal case where the error term is zero, β 0 represents the value of the dependent variable y at x = 0. Within the range of the independent variable x, β 1 represents the increase in the outcome y corresponding...

Coefficients of correlation and determination

In this section, we will discuss two related notions – coefficients of correlation and coefficients of determination.

Coefficients of correlation

A coefficient of correlation is a measure of the statistical linear relationship between two variables and can be computed using the following formula:

r =  1 _ n − 1 Σ i=1 n (x i −  ‾ x  _ s x )(y i −  ‾ y  _ s y )

The reader can go here – https://shiny.rit.albany.edu/stat/corrsim/ – to simulate the correlation relationship between two variables.

Figure 6.3 – Simulated bivariate distribution

By observing the scatter plots, we can see the direction and the strength of the linear relationship between the two variables and their outliers. If the direction is positive (r>0), then...

Required model assumptions

Like the parametric tests we discussed in Chapter 4, Parametric Tests, linear regression is a parametric method and requires certain assumptions to be met for the results to be valid. For linear regression, there are four assumptions:

• A linear relationship between variables
• The normality of the residuals
• The homoscedasticity of the residuals
• Independent samples

Let’s discuss each of these assumptions individually.

A linear relationship between the variables

When thinking about fitting a linear model to data, our first consideration should be whether the model is appropriate for the data. When working with two variables, the relationship between the variables should be assessed with a scatter plot. Let’s look at an example. Three scatter plots are shown in Figure 6.6. The data is plotted, and the actual function used to generate the data is drawn over the data points. The leftmost plot shows data exhibiting a...

Testing for significance and validating models

Up to this point in the chapter, we have discussed the concepts of the OLS approach to linear regression modeling; the coefficients in a linear model; the coefficients of correlation and determination; and the assumptions required for modeling with linear regression. We will now begin our discussion on testing for significance and model validation.

Testing for significance

To test for significance, let us load statsmodels macrodata data set so we can build a model that tests the relationship between real gross private domestic investment, realinv, and real private disposable income, realdpi:

import numpy as np
import pandas as pd
import seaborn as sns
import statsmodels.api as sm
import matplotlib.pyplot as plt
from statsmodels.nonparametric.smoothers_lowess import lowess
df = sm.datasets.macrodata.load().data

Least squares regression requires a constant coefficient in order to derive the intercept. In the least squares equation...

Summary

In this chapter, we discussed an overview of simple linear regression between one explanatory variable and one response variable. The topics we covered include the following:

• The OLS method for simple linear regression
• Coefficients of correlation and determination and their calculations and significance
• The assumptions required for least squares regression
• Methods of analysis for model and parameter significance
• Model validation

We looked closely at the concept of the square of error and how the sum of squared errors is meaningful for building and validating linear regression models. Then, we walked through the four pertinent assumptions required to make linear regression a stable solution. After, we provided an overview of four diagnostic plots and their interpretations with respect to assessing the presence of various issues related to heteroscedasticity, linearity, outliers, and serial correlation. We then walked through an example of using...