Improving Your RNN Performance

This chapter goes through some techniques for improving your recurrent neural network model. Often, the initial results from your model can be disappointing, so you need to find ways of improving them. This can be done with various methods and tools, but we will focus on two main areas:

Improving the RNN model performance with data and tuning
Optimizing the TensorFlow library for better results

First, we will see how more data, as well as tuning the hyperparameters, can yield significantly better results. Then our focus will shift to getting the most out of the built-in TensorFlow functionality. Both approaches are applicable to any task that involves the neural network model, so the next time you want to do image recognition with convolutional networks or fix a rescaled image with GAN, you can apply the same techniques for perfecting your model...

Improving your RNN model

When working on a problem using RNN (or any other network), your process looks like this:

First, you come up with an idea for the model, its hyperparameters, the number of layers, how deep the network should be, and so on. Then the model is implemented and trained in order to produce some results. Finally, these results are assessed and the necessary modifications are made. It is rarely the case that you'll receive meaningful results from the first run. This cycle may occur multiple times until you are satisfied with the outcome.

Considering this approach, one important question comes to mind: How can we change the model so the next cycle produces better results?

This question is tightly connected to your understanding of the network's results. Let's discuss that now.

As you already know, in the beginning of each...

Optimizing the TensorFlow library

This section focuses mostly on practical advice that can be directly implemented in your code. The TensorFlow team has provided a large set of tools that can be utilized to improve your performance. These techniques are constantly being updated to achieve better results. I strongly recommend watching TensorFlow's video on training performance from the 2018 TensorFlow conference (https://www.youtube.com/watch?v=SxOsJPaxHME). This video is accompanied by nicely aggregated documentation, which is also a must-read (https://www.tensorflow.org/performance/).

Now, let's dive into more details around what you can do to achieve faster and more reliable training.

Let's first start with an illustration from TensorFlow that presents the general steps of training a neural network. You can divide this process into three...

Summary

In this chapter, we covered a lot of new and exciting approaches for optimizing your model's performance, both on a general level, and specifically, using the TensorFlow library.

The first part covered techniques for improving your RNN performance by selecting, processing, and transforming your data, as well as tuning your hyperparameters. You also learned how to understand your model in more depth, and now know what should be done to make it work better.

The second part was specifically focused on practical ways of improving your model's performance using the built-in TensorFlow functions. The team at TensorFlow seeks to make it as easy as possible for you to quickly achieve the results you want by providing distributed environments and optimization techniques with just a few lines of code.

Combining both of the techniques covered in this...