Going Deeper – The Mechanics of PyTorch

In Chapter 12, Parallelizing Neural Network Training with PyTorch, we covered how to define and manipulate tensors and worked with the torch.utils.data module to build input pipelines. We further built and trained a multilayer perceptron to classify the Iris dataset using the PyTorch neural network module (torch.nn).

Now that we have some hands-on experience with PyTorch neural network training and machine learning, it’s time to take a deeper dive into the PyTorch library and explore its rich set of features, which will allow us to implement more advanced deep learning models in upcoming chapters.

In this chapter, we will use different aspects of PyTorch’s API to implement NNs. In particular, we will again use the torch.nn module, which provides multiple layers of abstraction to make the implementation of standard architectures very convenient. It also allows us to implement custom NN layers, which is very useful...

The key features of PyTorch

In the previous chapter, we saw that PyTorch provides us with a scalable, multiplatform programming interface for implementing and running machine learning algorithms. After its initial release in 2016 and its 1.0 release in 2018, PyTorch has evolved into one of the two most popular frameworks for deep learning. It uses dynamic computational graphs, which have the advantage of being more flexible compared to its static counterparts. Dynamic computational graphs are debugging friendly: PyTorch allows for interleaving the graph declaration and graph evaluation steps. You can execute the code line by line while having full access to all variables. This is a very important feature that makes the development and training of NNs very convenient.

While PyTorch is an open-source library and can be used for free by everyone, its development is funded and supported by Facebook. This involves a large team of software engineers who expand and improve the library...

PyTorch’s computation graphs

PyTorch performs its computations based on a directed acyclic graph (DAG). In this section, we will see how these graphs can be defined for a simple arithmetic computation. Then, we will see the dynamic graph paradigm, as well as how the graph is created on the fly in PyTorch.

Understanding computation graphs

PyTorch relies on building a computation graph at its core, and it uses this computation graph to derive relationships between tensors from the input all the way to the output. Let’s say that we have rank 0 (scalar) tensors a, b, and c and we want to evaluate z = 2 × (a – b) + c.

This evaluation can be represented as a computation graph, as shown in Figure 13.1:

Figure 13.1: How a computation graph works

As you can see, the computation graph is simply a network of nodes. Each node resembles an operation, which applies a function to its input tensor or tensors...

PyTorch tensor objects for storing and updating model parameters

We covered tensor objects in Chapter 12, Parallelizing Neural Network Training with PyTorch. In PyTorch, a special tensor object for which gradients need to be computed allows us to store and update the parameters of our models during training. Such a tensor can be created by just assigning requires_grad to True on user-specified initial values. Note that as of now (mid-2021), only tensors of floating point and complex dtype can require gradients. In the following code, we will generate tensor objects of type float32:

>>> a = torch.tensor(3.14, requires_grad=True)
>>> print(a)
tensor(3.1400, requires_grad=True)
>>> b = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
>>> print(b)
tensor([1., 2., 3.], requires_grad=True)

Notice that requires_grad is set to False by default. This value can be efficiently set to True by running requires_grad_().

method_() is an in...

Computing gradients via automatic differentiation

As you already know, optimizing NNs requires computing the gradients of the loss with respect to the NN weights. This is required for optimization algorithms such as stochastic gradient descent (SGD). In addition, gradients have other applications, such as diagnosing the network to find out why an NN model is making a particular prediction for a test example. Therefore, in this section, we will cover how to compute gradients of a computation with respect to its input variables.

Computing the gradients of the loss with respect to trainable variables

PyTorch supports automatic differentiation, which can be thought of as an implementation of the chain rule for computing gradients of nested functions. Note that for the sake of simplicity, we will use the term gradient to refer to both partial derivatives and gradients.

Partial derivatives and gradients

A partial derivative can be understood as the rate of change...

Simplifying implementations of common architectures via the torch.nn module

You have already seen some examples of building a feedforward NN model (for instance, a multilayer perceptron) and defining a sequence of layers using the nn.Module class. Before we take a deeper dive into nn.Module, let’s briefly look at another approach for conjuring those layers via nn.Sequential.

Implementing models based on nn.Sequential

With nn.Sequential (https://pytorch.org/docs/master/generated/torch.nn.Sequential.html#sequential), the layers stored inside the model are connected in a cascaded way. In the following example, we will build a model with two densely (fully) connected layers:

>>> model = nn.Sequential(
...     nn.Linear(4, 16),
...     nn.ReLU(),
...     nn.Linear(16, 32),
...     nn.ReLU()
... )
>>> model
Sequential(
  (0): Linear(in_features=4, out_features=16, bias=True)
  (1): ReLU()
  (2): Linear(in_features=16, out_features=32, bias=True)
  (3...

Project one – predicting the fuel efficiency of a car

So far, in this chapter, we have mostly focused on the torch.nn module. We used nn.Sequential to construct the models for simplicity. Then, we made model building more flexible with nn.Module and implemented feedforward NNs, to which we added customized layers. In this section, we will work on a real-world project of predicting the fuel efficiency of a car in miles per gallon (MPG). We will cover the underlying steps in machine learning tasks, such as data preprocessing, feature engineering, training, prediction (inference), and evaluation.

Working with feature columns

In machine learning and deep learning applications, we can encounter various different types of features: continuous, unordered categorical (nominal), and ordered categorical (ordinal). You will recall that in Chapter 4, Building Good Training Datasets – Data Preprocessing, we covered different types of features and learned how to handle each...

Project two – classifying MNIST handwritten digits

For this classification project, we are going to categorize MNIST handwritten digits. In the previous section, we covered the four essential steps for machine learning in PyTorch in detail, which we will need to repeat in this section.

You will recall that in Chapter 12 you learned the way of loading available datasets from the torchvision module. First, we are going to load the MNIST dataset using the torchvision module.

1. The setup step includes loading the dataset and specifying hyperparameters (the size of the train set and test set, and the size of mini-batches):

   >>> import torchvision
   >>> from torchvision import transforms
   >>> image_path = './'
   >>> transform = transforms.Compose([
   ...     transforms.ToTensor()
   ... ])
   >>> mnist_train_dataset = torchvision.datasets.MNIST(
   ...     root=image_path, train=True,
   ...     transform=transform, download...

Higher-level PyTorch APIs: a short introduction to PyTorch-Lightning

In recent years, the PyTorch community developed several different libraries and APIs on top of PyTorch. Notable examples include fastai (https://docs.fast.ai/), Catalyst (https://github.com/catalyst-team/catalyst), PyTorch Lightning (https://www.pytorchlightning.ai), (https://lightning-flash.readthedocs.io/en/latest/quickstart.html), and PyTorch-Ignite (https://github.com/pytorch/ignite).

In this section, we will explore PyTorch Lightning (Lightning for short), which is a widely used PyTorch library that makes training deep neural networks simpler by removing much of the boilerplate code. However, while Lightning’s focus lies in simplicity and flexibility, it also allows us to use many advanced features such as multi-GPU support and fast low-precision training, which you can learn about in the official documentation at https://pytorch-lightning.rtfd.io/en/latest/.

There is also a bonus introduction...

Summary

In this chapter, we covered PyTorch’s most essential and useful features. We started by discussing PyTorch’s dynamic computation graph, which makes implementing computations very convenient. We also covered the semantics of defining PyTorch tensor objects as model parameters.

After we considered the concept of computing partial derivatives and gradients of arbitrary functions, we covered the torch.nn module in more detail. It provides us with a user-friendly interface for building more complex deep NN models. Finally, we concluded this chapter by solving a regression and classification problem using what we have discussed so far.

Now that we have covered the core mechanics of PyTorch, the next chapter will introduce the concept behind convolutional neural network (CNN) architectures for deep learning. CNNs are powerful models and have shown great performance in the field of computer vision.

Join our book’s Discord space

Join our Discord...