Computational Graphs and Linear Models

By now you should have an understanding of the theory of linear models and neural networks, as well as a knowledge of the fundamentals of PyTorch. In this chapter, we will be putting all this together by implementing some ANNs in PyTorch. We will focus on the implementation of linear models, and show how they can be adapted to perform multi-class classification. We will discuss the following topics in relation to PyTorch:

• autograd
• Computational graphs
• Linear regression
• Logistic regression
• Multi-class classification

autograd

As we saw in the last chapter, much of the computational work for ANNs involves calculating derivatives to find the gradient of the cost function. PyTorch uses the autograd package to perform automatic differentiation of operations on PyTorch tensors. To see how this works, let's look at an example:

In the preceding code, we create a 2 x 3 torch tensor and, importantly, set the requires_grad attribute to True. This enables the calculation of gradients across subsequent operations. Notice also that we set the dtype to torch.float, since this is the data type that PyTorch uses for automatic differentiation. We perform a sequence of operations and then take the mean of the result. This returns a tensor containing a single scalar. This is normally what autograd requires to calculate the gradient of the preceding operations. This could be any sequence of operations;...

Linear models

Linear models are an essential way to understand the mechanics of ANNs. Linear regression is used to both predict a continuous variable and also, in the case of logistic regression for classification, to predict a class. Neural networks are extremely useful for multi-class classification, since their architecture can be naturally adapted to multiple inputs and outputs.

Linear regression in PyTorch

Let's see how PyTorch implements a simple linear network. We could use autograd and backward to manually iterate through gradient descent. This unnecessarily low-level approach encumbers us with a lot of code that will be difficult to maintain, understand, and upgrade. Fortunately, PyTorch has a very straightforward...

Multi-class classification example

So far, we have been using trivial examples to demonstrate core concepts in PyTorch. We are now ready to explore a more real-world example. The dataset we will be using is the MNIST dataset of hand-written digits from 0 to 9. The task is to correctly identify each sample image with the correct digit.

The classification model we will be building consists of several layers and these are outlined in the following diagram:

The images we are working with are 28 x 28 pixels in size, and each pixel in each image is characterized by a single number, indicating its gray scale. This is why we need 28 x 28 or 784 inputs to the model. The first layer is a linear layer with 10 outputs, one output for each label. These outputs are fed into to the softmax activation layer and cross-entropy loss layer. The 10 output dimensions represent the 10 possible classes...

Summary

In this chapter, we have explored linear models and applied them to the tasks of linear regression, logistic regression, and multi-class classification. We have seen how autograd calculates gradients and how PyTorch works with computational graphs. The multi-class classification model we built did a reasonable job of predicting hand-written digits; however, its performance is far from optimal. The best deep learning models are able to get near 100% accuracy on this dataset.

We will see in Chapter 4, Convolutional Networks, how adding more layers and using convolutional networks can improve performance.