Generating Your Own Book Chapter

In this chapter, we will take a step further into exploring the TensorFlow library and how it can be leveraged to solve complex tasks. In particular, you will build a neural network that generates a new (non-existing) chapter of a book by learning patterns from the existing chapters. In addition, you will grasp more of the TensorFlow functionalities, such as saving/restoring a model, and so on. 

This chapter will also introduce a new and more powerful recurrent neural network model called the gated recurrent unit (GRU). You will learn how it works and why we are choosing it over the simple RNN. 

In summary, the topics of the chapter include the following:

• Why use the GRU network? You will learn how the GRU network works, what problems it solves, and what its benefits are.
• Generating your book chapter—you will go step by step...

Why use the GRU network?

In recent years, the recurrent neural network model has presented fascinating results which can even be seen in real-life applications like language translation, speech synthesis and more. A phenomenal application of GRUs happens to be text generation. With the current state-of-the-art models, we can see results which, a decade ago, were just a dream. If you want to truly appreciate these results, I strongly recommend you read Andrej Karpathy's article on The Unreasonable Effectiveness of Recurrent Neural Networks (http://karpathy.github.io/2015/05/21/rnn-effectiveness/). 

Having said that, we can introduce the Gated Recurrent Unit (GRU) as a model which sits behind these exceptional outcomes. Another model of that kind is the Long Short-Term Memory (LSTM) which is slightly more advanced. Both architectures aim to...

Generating your book chapter

After going through the theoretical part of this chapter, we are ready to dive into coding. I hope you grasp the fundamental behind the GRU model and will feel comfortable seeing the notations in the TensorFlow program. It consists of five parts, most of which may be familiar to you from Chapter 2, Building Your First RNN with TensorFlow:

• Obtaining the book text: this one is really straightforward. Your task is to assure a lot of plain text is ready for training. 
• Encoding the text: this one can be challenging, since we need to accommodate the encoding with the proper dimensions. Sometimes, this operation can take more time than expected but it is a requirement for compiling the program flawlessly. There are different types of encoding algorithms and we will choose a fairly simple one so you fully understand its true essence.
• Building the TensorFlow...

Summary

 

In this chapter, you went through the process of building a book chapter generator using a Gated Recurrent Unit neural network. You understood what sits behind this powerful model and how you can put it into practice with a handful of lines of code using TensorFlow. In addition, you faced the challenge of preparing and clearing your data so that your model is trained correctly. 

In the next chapter, you will fortify your skills by implementing your first real-life practical application—a language translator. You have probably faced the online Google Translate software and were amazed by how well it worked. In the next chapter, you will understand what sits behind a sophisticated system like that and why its level of accuracy has increased drastically in recent years. 

I hope the current chapter advanced your deep learning knowledge and that you...

External links

• The Unreasonable Effectiveness of Recurrent Neural Networks—http://karpathy.github.io/2015/05/21/rnn-effectiveness/
• Sigmoid activation function—https://www.youtube.com/watch?v=WcDtwxi7Ick&t=3s
• The difference between sigmoid and tanh activation function—https://stats.stackexchange.com/questions/101560/tanh-activation-function-vs-sigmoid-activation-function
• Element-wise multiplication—https://www.youtube.com/watch?v=2GPZlRVhQWY
• Learning rate in the neural network—https://towardsdatascience.com/understanding-learning-rates-and-how-it-improves-performance-in-deep-learning-d0d4059c1c10
• The number of hidden units in TensorFlow—https://stackoverflow.com/questions/37901047/what-is-num-units-in-tensorflow-basiclstmcell
• The formula for calculating the number of hidden units—https://stats.stackexchange.com/questions/181...