Linear Neural Networks

In this chapter, we will go over some of the concepts in machine learning. It is expected that you have previously studied and have an understanding of machine learning. So this chapter will serve as a refresher for some of the concepts that will be needed throughout this book, rather than a comprehensive study of all the machine learning approaches.

In this chapter, we will focus on linear neural networks, which are the simplest type of neural networks and are used for tasks such as linear regression, polynomial regression, logistic regression, and softmax regression, which are used most frequently in statistical learning.

We use regression to explain the relationship between one or more independent variables and a dependent variable. The concepts we will learn in this chapter are crucial for furthering our understanding of how machine learning works before...

Linear regression

The purpose of regression is to find the relationship that exists between data (denoted by x) and its corresponding output (denoted by y) and predict it. The output of all regression problems is a real number (). This can be applied to a range of problems, such as predicting the price of a house or what rating a movie will have.

In order for us to make use of regression, we need to use the following:

• Input data, which could be either scalar values or vectors. This is sometimes referred to as features.
• Training examples, which include a good number of (x_i, y_i) pairs; that is, the output for each input.
• A function that captures the relationship between the input and output—the model.
• A loss or an objective function, which tells us how accurate our model is.
• Optimization, to minimize the loss or the objective function.

Before we go further, let's...

Polynomial regression

Linear regression, as you might imagine, isn't a one-size-fits-all solution that we can use for any problem. A lot of the relationships that exist between variables in the real world are not linear; that is, a straight line isn't able to capture the relationship. For these problems, we use a variant of the preceding linear regression known as polynomial regression, which can capture more complexities, such as curves. This method makes use of applying different powers to the explanatory variable to discover non-linear problems. This looks as follows:

Or, we could have the following:

This is the case for .

As you can see from the preceding equation, a model such as this is not only able to capture a straight line (if needed) but can also generate a second-order, third-order, or n^th-order equation that fits the data points.

Let's suppose we...

Logistic regression

There is another kind of regression that we often use in practice—logistic regression. Suppose we want to determine whether or not an email is spam. In this case, our x(s) value could be occurrences of !(s) or the total number of spelling errors in the email. Then, y can take on the value of 1 (for spam) and 0 (for not spam).

In this kind of case, linear regression will simply not work since we are not predicting a real value—we are trying to predict which class the email belongs to.

This will usually end up looking as follows:

As you can see, the data is grouped into two areas—one that represents non-spam and another that represents spam.

We can calculate this as follows:

Here, .

However, this only works for binary classification. What if we want to classify multiple classes? Then, we can use softmax regression, which is an extension...

Summary

In this chapter, we learned about various forms of regression, such as (multiple) linear regression, polynomial regression, logistic regression, and softmax regression. Each of these models has aided us in figuring out the relationship that exists between one or more independent variable(s) and a dependent variable. For some of you, these concepts may seem very rudimentary, but they will serve us well on our journey throughout this book and in gaining a deeper understanding of the concepts to come.

In the next chapter, we will learn about feedforward neural networks.