Getting the Most out of PyTorch

By now, you should be able to build and train three different types of model: linear, convolutional, and recurrent. You should have an appreciation of the theory and mathematics behind these model architectures and explain how they make predictions. Convolutional networks are probably the most studied deep learning network, especially in relation to image data. Of course, both convolutional and recurrent networks make extensive use of linear layers, so the theory behind linear networks, most notably linear regression and gradient descent, is fundamental to all artificial neural networks.

Our discussion so far has been fairly contained. We have looked at a well-studied problem, such as classification using MNIST, to give you a solid understanding of the basic PyTorch building blocks. This final chapter is the launching pad for your use of PyTorch...

Multiprocessor and distributed environments

There are a variety of multiprocessor and distributed environment possibilities. The most common reason for using more than one processor is, of course, to make models run faster. The time it takes to load MNIST—a relatively tiny dataset of 60,000 images—to memory is not significant. However, consider the situation where we have giga or terabytes of data, or if the data is distributed across multiple servers. The situation is even more complex when we consider online models, where data is being harvested from multiple servers in real time. Clearly, some sort of parallel processing capability is required.

Using a GPU

The simplest way to make a model run faster is to...

Optimization techniques

The torch.optim package contains a number of optimization algorithms, and each of these algorithms has several parameters that we can use to fine-tune deep learning models. Optimization is a critical component in deep learning, so it is no surprise that different optimization techniques can be key to a model's performance. Remember, its role is to store and update the parameter state based on the calculated gradients of the loss function.

Optimizer algorithms

There are a number of optimization algorithms besides SGD available in PyTorch. The following code shows one such algorithm:

optim.Adadelta(params, lr=1.0, rho=0.9, eps=1e-06, weight_decay=0)

The Adedelta algorithm is based on stochastic gradient...

Pretrained models

One of the major difficulties with image classification models is the lack of labeled data. It is difficult to assemble a labeled dataset of sufficient size to train a model well; it is an extremely time consuming and laborious task. This is not such a problem for MNIST, since the images are relatively simple. They are greyscale and largely consist only of target features, there are no distracting background features, and the images are all aligned the same way and are of the same scale. A small dataset of 60,000 images is quite sufficient to train a model well. It is rare to find such a well-organized and consistent dataset in the problems we encounter in real life. Images are often of variable quality, and the target features can be obscured or distorted. They can also be of widely variable scales and rotations. The solution is to use a model architecture that...

Summary

Now that you have an understanding of the foundations of deep learning, you should be well placed to apply this knowledge to specific learning problems that you are interested in. In this chapter, we have developed an out-of-the-box solution for image classification using pretrained models. As you have seen, this is quite simple to implement, and can be applied to almost any image classification problem you can think of. Of course, the actual performance in each situation will depend on the number and quality of images presented, as well as the precise tuning of the hyperparameters associated with each model and task.

You can generally get very good results on most image classification tasks by simply running the pretrained models with default parameters. This requires no theoretical knowledge, apart from installing the programs' running environment. You will find...