Improving Inference Performance with MXNet

In previous chapters, we leveraged MXNet’s capabilities to solve computer vision and natural language processing tasks. In those chapters, the focus was on obtaining the maximum performance out of pre-trained models, leveraging the Model Zoo API from GluonCV and GluonNLP. We trained these models using different approaches from scratch, including transfer learning and fine-tuning. In the previous chapter, we explored how some advanced techniques can be leveraged to optimize the training process. Finally, in this chapter, we will focus on improving the performance of the inference process itself, accelerating how we can obtain results from our models with several topics related to edge AI computing.

To achieve the objective of optimizing the performance of our inference pipeline, MXNet contains different features. We have already briefly discussed some of those features, such as the concept of Automatic Mixed Precision (AMP), which...

Technical requirements

Apart from the technical requirements specified in the Preface, the following apply:

• Ensure that you have completed Recipe 1, Installing MXNet, from Chapter 1, Up and Running with MXNet.
• Ensure that you have completed Chapter 5, Analyzing Images with Computer Vision, and Chapter 6, Understanding Text with Natural Language Processing.
• Ensure that you have completed Chapter 7, Optimizing Models with Transfer Learning and Fine-Tuning.

The code for this chapter can be found at the following GitHub URL: https://github.com/PacktPublishing/Deep-Learning-with-MXNet-Cookbook/tree/main/ch09.

Furthermore, you can access each recipe directly from Google Colab, for example, for the first recipe of this chapter: https://github.com/PacktPublishing/Deep-Learning-with-MXNet-Cookbook/blob/main/ch09/9_1_Introducing_inference_optimization_features.ipynb.

Introducing inference optimization features

In the previous chapters, we have seen how we can leverage MXNet, GluonCV, and GluonNLP to retrieve pre-trained models in certain datasets (such as ImageNet, MS COCO, or IWSLT2015) and use them for our specific tasks and datasets. Furthermore, we used transfer learning and fine-tuning techniques to improve the algorithmic performance of those tasks/datasets.

In this recipe, we will introduce (and revisit) several concepts and features that will optimize our inference loops to improve our runtime performance, and we will analyze the trade-offs involved.

Getting ready

As in previous chapters, in this recipe, we will be using some matrix operations and linear algebra, but it will not be hard at all.

How to do it...

In this recipe, we will be carrying out the following steps:

1. Hybridizing our models
2. Applying float16 and AMP for inference
3. Applying quantization by using INT8
4. Profiling our models

Let’...

Optimizing inference for image segmentation

In the previous recipe, we saw how we can leverage MXNet and Gluon to optimize the inference of our models, applying different techniques, such as improving the runtime performance using hybridization; how using half-precision (float16) in combination with AMP can strongly reduce our inference times; and how to take advantage of further optimizations with data types such as Int8 quantization.

Now, we can revisit a problem we have been working with throughout the book, image segmentation. We have worked with this task in recipes from previous chapters. In Recipe 4, Segmenting objects semantically with MXNet Model Zoo – PSPNet and DeepLabv3, from Chapter 5, Analyzing Images with Computer Vision, we introduced the task and the datasets that we will be using in this recipe, MS COCO and Penn-Fudan Pedestrian, and learned how to use pre-trained models from GluonCV Model Zoo.

Furthermore, in Recipe 3, Improving performance for segmenting...

Optimizing inference when translating text from English to German

In the initial recipe, we saw how we can leverage MXNet and Gluon to optimize the inference of our models, applying different techniques: improving the runtime performance using hybridization; how using half-precision (float16) in combination with AMP can strongly reduce our inference times; and how to take advantage of further optimizations with data types such as Int8 quantization.

Now, we can revisit a problem we have been working with throughout the book: translating English to German. We have worked with translation tasks in recipes from previous chapters. In Recipe 4, Translating text from Vietnamese to English, from Chapter 6, Understanding Text with Natural Language Processing, we introduced the task of translating text, while also learning how to use pre-trained models from GluonCV Model Zoo.

Furthermore, in Recipe 4, Improving performance for translating English to German, from Chapter 7, Optimizing Models...