Building Your First RNN with TensorFlow

In this chapter, you will gain a hands-on experience of building a recurrent neural network (RNN). First, you will be introduced to the most widely used machine learning library—TensorFlow. From learning the basics to advancing into some fundamental techniques, you will obtain a reasonable understanding of how to apply this powerful library to your applications. Then, you will take on a fairly simple task of building an actual model. The process will show you how to prepare your data, train the network, and make predictions.

In summary, the topics of this chapter include the following:

• What are you going to build?: Introduction of your task
• Introduction to TensorFlow: Taking first steps into learning the TensorFlow framework
• Coding the RNN: You will go through the process of writing your first neural network using TensorFlow. This...

What are you going to build?

 

Your first steps into the practical world of recurrent neural networks will be to build a simple model which determines the parity (http://mathworld.wolfram.com/Parity.html) of a bit sequence . This is a warm-up exercise released by OpenAI in January 2018 (https://blog.openai.com/requests-for-research-2/). The task can be explained as follows: 

Given a binary string of a length of 50, determine whether there is an even or odd number of ones. If that number is even, output 0, otherwise 1.

Later in this chapter, we will give a detailed explanation of the solution, together with addressing the difficult parts and how to tackle them.

Introduction to TensorFlow

TensorFlow is an open source library built by Google, which aims to assist developers in creating machine learning models of any kind. The recent improvements in the deep learning space created the need for an easy and fast way of building neural networks. TensorFlow addresses this problem in an excellent fashion, by providing a wide range of APIs and tools to help developers focus on their specific problem, rather than dealing with mathematical equations and scalability issues. 

TensorFlow offers two main ways of programming a model:

• Graph-based execution
• Eager execution

Graph-based execution

Graph-based execution is an alternative way of representing mathematical equations and functions...

Coding the recurrent neural network

As mentioned before, the aim of our task is to build a recurrent neural network that predicts the parity of a bit sequence. We will approach this problem in a slightly different way. Since the parity of a sequence depends on the number of ones, we will sum up the elements of the sequence and find whether the result is even or not. If it is even, we will output 0, otherwise, 1. 

This section of the chapter includes code samples and goes through the following steps:

• Generating data to train the model
• Building the TensorFlow graph (using TensorFlow's built-in functions for recurrent neural networks)
• Training the neural network with the generated data
• Evaluating the model and determining its accuracy

Generating data

...

Summary

In this chapter, you explored how to build a simple recurrent neural network to solve the problem of identifying sequence parity. You obtained a brief understanding of the TensorFlow library and how it can be utilized for building deep learning models. I hope the study of this chapter leaves you more confident in your deep learning knowledge, as well as excited to learn and grow more in this field. 

In the next chapter, you will go a step further by implementing a more sophisticated neural network for the task of generating text. You will gain both theoretical and practical experience. This will result in you learning about a new type of network, GRU, and understanding how to implement it in TensorFlow. In addition, you will face the challenge of formatting your input text correctly as well as using it for training the TensorFlow graph. 

I can assure you that...

External links

• Parity: http://mathworld.wolfram.com/Parity.html
• Request for Research 2.0 by OpenAI: https://blog.openai.com/requests-for-research-2/ 
• Eager execution documentation: https://www.tensorflow.org/guide/eager
• Eager execution (TensorFlow Conference 2018): https://www.youtube.com/watch?v=T8AW0fKP0Hs
• Python installation: https://www.python.org/downloads/
• Understanding num_hidden_units : https://stackoverflow.com/questions/37901047/what-is-num-units-in-tensorflow-basiclstmcell
• Adam versus Gradient descent optimizer: https://stats.stackexchange.com/questions/184448/difference-between-gradientdescentoptimizer-and-adamoptimizer-tensorflow
• Understanding sess.run(tf.global_variables_initializer(): https://stackoverflow.com/questions/44433438/understanding-tf-global-variables-initializer

...