What options do we have?
Once we have decided to accelerate the training process of a model, we can take two directions, as illustrated in Figure 2.1:
Figure 2.1 – Approaches to accelerating the training phase
In the first option (Modify the software stack), we go through each layer of the software stack used to train a model to seek opportunities to improve the training process. In simpler words, we can change the application code, install and use a specialized library, or enable a special capability regarding the operating system or container environment.
This first approach relies on having profound knowledge of performance tuning techniques. In addition, it demands a high sense of investigation to identify bottlenecks and apply the most suitable solution to overcome them. Thus, this approach is about harnessing the most hardware and software resources by extracting the maximum performance of the computing system.
Nevertheless, remark that...
