Simplifying the Model

Have you heard about parsimony? Parsimony, in the context of model estimation, concerns keeping a model as simple as possible. Such a principle comes from the assumption that complex models (models with a higher number of parameters) overfit the training data, thus reducing the capacity to generalize and make good predictions.

In addition, simplifying neural networks has two main benefits: reducing the model training time and making the model feasible to run in resource-constrained environments. One of the approaches to simplifying a model relies on reducing the number of parameters of the neural network by employing pruning and compression techniques.

In this chapter, we show how to simplify a model by reducing the number of parameters of the neural network without sacrificing its quality.

Here is what you will learn as part of this chapter:

• The key benefits of simplifying a model
• The concept and techniques of model pruning and compression...

Technical requirements

You can find the complete code of examples mentioned in this chapter in the book’s GitHub repository at https://github.com/PacktPublishing/Accelerate-Model-Training-with-PyTorch-2.X/blob/main.

You can access your favorite environments to execute this notebook, such as Google Colab or Kaggle.

Knowing the model simplifying process

In simpler words, simplifying a model concerns removing connections, neurons, or entire layers of the neural network to get a lighter model, i.e., a model with a reduced number of parameters. Naturally, the efficiency of the simplified version must be very close to the one achieved by the original model. Otherwise, simplifying the model does not make any sense.

To understand this topic, we must answer the following questions:

• Why simplify a model? (reason)
• How do we simplify a model? (process)
• When do we simplify a model? (moment)

We will go through each of these questions in the following sections to get an overall understanding of model simplification.

Note

Before moving on in this chapter, it is essential to say that model simplification is still an open research area. Consequently, some concepts and terms cited in this book may differ a little bit from other materials or how they are employed on frameworks and...

Using Microsoft NNI to simplify a model

Neural Network Intelligence (NNI) is an open-source project created by Microsoft to help deep learning practitioners automate tasks such as hyperparameter automatization and neural architecture searches.

NNI also has a set of tools to deal with model simplification in a simpler and straightforward manner. So, we can easily simplify a model by adding a couple of lines to our original code. NNI supports PyTorch and other well-known deep learning frameworks such as TensorFlow.

Note

PyTorch has its own API to prune models, namely torch.prune. Unfortunately, at the time of writing this book, this API does not provide a mechanism to compress a model. Therefore, we have decided to introduce NNI as the solution to accomplish this task. More information about NNI can be found at https://github.com/microsoft/nni.

Let’s start by getting an overview of NNI in the next section.

Overview of NNI

Because NNI is not a native component...

Quiz time!

Let’s review what we have learned in this chapter by answering a few questions. At first, try to answer these questions without consulting the material.

Note

The answers to all these questions are available at https://github.com/PacktPublishing/Accelerate-Model-Training-with-PyTorch-2.X/blob/main/quiz/chapter06-answers.md.

Before starting the quiz, remember that it is not a test at all! This section aims to complement your learning process by revising and consolidating the content covered in this chapter.

Choose the correct option for the following questions.

1. What are the two steps to take when simplifying a workflow?
   1. Reduction and compression.
   2. Pruning and reduction.
   3. Pruning and compression.
   4. Reduction and zipping.
2. A pruning technique usually has the following dimensions:
   1. Criterion, scope, and method.
   2. Algorithm, scope, and magnitude.
   3. Criterion, constraints, and targets.
   4. Algorithm, constraints, and targets.
3. Concerning the compression phase, we can...

Summary

In this chapter, you learned that simplifying a model by reducing the number of parameters can accelerate the network training process, besides making the model feasible to run on resource-constrained platforms.

Then, we saw that the simplification process consists of two phases: pruning and compression. The former is responsible for determining which parameters must be dropped off from the network, whereas the latter effectively removes the parameters from the model.

Although PyTorch provides an API to prune the model, it is not fully useful to simplify a model. Thus, you were introduced to Microsoft NNI, a powerful toolkit to automate tasks related to deep learning modes. Among the features provided by NNI, this tool offers a complete workflow to simplify a model. All of this is achieved with a couple of new lines added to the original code.

In the next chapter, you will learn how to reduce the numeric precision adopted by the neural network to accelerate the training...