Distributed Training at a Glance

When we face a complex problem in real life, we usually try to solve it by dividing the big problem into small parts that are easier to treat. So, by combining the partial solutions obtained from the small pieces of the original problem, we reach the final solution. This strategy, called divide and conquer, is frequently used to solve computational tasks. We can say that this approach is the basis of the parallel and distributed computing areas.

It turns out that this idea of dividing a big problem into small pieces comes in handy to accelerate the training process of complex models. In cases where using a single resource is not enough to train the model in a reasonable time, the unique way out relies on breaking down the training process and spreading it across multiple resources. In other words, we need to distribute the training process.

Here is what you will learn as part of this chapter:

• The basic concepts of distributed training
...

Technical requirements

You can find the complete code examples mentioned in this chapter in the book’s GitHub repository at https://github.com/PacktPublishing/Accelerate-Model-Training-with-PyTorch-2.X/blob/main.

You can access your favorite environment to execute the code provided, such as Google Colab or Kaggle.

A first look at distributed training

We’ll start this chapter by discussing the reasons for distributing the training process among multiple resources. Then, we’ll learn what resources are commonly used to execute this process.

When do we need to distribute the training process?

The most common reason to distribute the training process concerns accelerating the building process. Suppose the training process is taking a long time to complete, and we have multiple resources at hand. In that case, we should consider distributing the training process among these various resources to reduce the training time.

The second motivation for going distributed is related to memory leaks to load a large model in a single resource. In this situation, we rely on distributed training to allocate different parts of the large model into distinct devices or resources so that the model can be loaded into the system.

However, distributed training is not a silver bullet that solves...

Learning the fundamentals of parallelism strategies

In the previous section, we learned that the distributed training approach divides the whole training process into small parts. As a result, the entire training process can be solved in parallel because each of these small parts is executed simultaneously in distinct computing resources.

The parallelism strategy defines how to divide the training process into small parts. There are two main parallelism strategies: model and data parallelism. The following sections explain both.

Model parallelism

Model parallelism divides the set of operations that are executed during the training process into smaller subsets of computing tasks. By doing this, the distributed process can run these smaller subsets of operations in distinct computing resources, thus accelerating the entire training process.

It turns out that operations executed in the forward and backward phases are not independent of each other. In other words, the execution...

Distributed training on PyTorch

This section introduces the basic workflow to implement distributed training on PyTorch, besides presenting the components used in this process.

Basic workflow

Generally speaking, the basic workflow to implement distributed training on PyTorch comprises the steps illustrated in Figure 8.14:

Figure 8.14 – Basic workflow to implement distributed training in PyTorch

Let’s look at each step in more detail.

Note

The complete code shown in this section is available at https://github.com/PacktPublishing/Accelerate-Model-Training-with-PyTorch-2.X/blob/main/code/chapter08/pytorch_ddp.py.

Initialize and destroy the communication group

The communication group is the logical entity that’s used by PyTorch to define and control the distributed environment. So, the first step to code the distributed training concerns initializing a communication group. This step is performed by instantiating an object...

Quiz time!

Let’s review what we have learned in this chapter by answering a few questions. Initially, try to answer these questions without consulting the material.

Note

The answers to all these questions are available at https://github.com/PacktPublishing/Accelerate-Model-Training-with-PyTorch-2.X/blob/main/quiz/chapter08-answers.md.

Before starting the quiz, remember that this is not a test! This section aims to complement your learning process by revising and consolidating the content covered in this chapter.

Choose the correct option for the following questions.

1. What are the two main reasons for distributing the training process?
   1. Reliability and performance improvement.
   2. Leak of memory and power consumption.
   3. Power consumption and performance improvement.
   4. Leak of memory and performance improvement.
2. Which are the two main parallel strategies to distribute the training process?
   1. Model and data parallelism.
   2. Model and hardware parallelism.
   3. Hardware and data parallelism...

Summary

In this chapter, you learned that distributed training is indicated to accelerate the training process and training models that do not fit on a device’s memory. Although going distributed can be a way out for both cases, we must consider applying performance improvement techniques before going distributed.

We can perform distributed training by adopting the model parallelism or data parallelism strategy. The former employs different paradigms to divide the model computation among multiple computing resources, while the latter creates model replicas to be trained over chunks of the training dataset.

We also learned that PyTorch relies on third-party components such as communication backends and program launchers to execute the distributed training process.

In the next chapter, we will learn how to spread out the distributed training process so that it can run on multiple CPUs located in a single machine.