Chapter 2. Understanding TensorFlow

In this chapter, you will get an in-depth understanding of TensorFlow. This is an open source distributed numerical computation framework, and it will be the main platform on which we will be implementing all our exercises.

We will get started with TensorFlow by defining a simple calculation and trying to compute it using TensorFlow. After we successfully complete this, we will investigate how TensorFlow executes this computation. This will help us to understand how the framework creates a computational graph to compute the outputs and execute this graph through something known as a session. Then we will gain a hands-on experience of the TensorFlow architecture by relating how TensorFlow executes things, with the help of an analogy of how a restaurant might operate.

Having gained a good conceptual and technical understanding of how TensorFlow operates, we will look at some of the important computational operations that the framework offers. First, we will...

What is TensorFlow?

In Chapter 1, Introduction to Natural Language Processing, we briefly discussed what TensorFlow is. Now let's take a closer look at it. TensorFlow is an open source distributed numerical computation framework released by Google that is mainly intended to alleviate the painful details of implementing a neural network (for example, computing derivatives of the weights of the neural network). TensorFlow takes this even a step further by providing efficient implementations of such numerical computations using Compute Unified Device Architecture (CUDA), which is a parallel computational platform introduced by NVIDIA. The Application Programming Interface (API) of TensorFlow at https://www.tensorflow.org/api_docs/python/ shows that TensorFlow provides thousands of operations that make our lives easier.

TensorFlow was not developed overnight. This is a result of the persistence of talented, good-hearted individuals who wanted to make a difference by bringing deep learning to...

Inputs, variables, outputs, and operations

Now with an understanding of the underlying architecture let's proceed to the most common elements that comprise a TensorFlow client. If you read any of the millions of TensorFlow clients available on the internet, they all (the TensorFlow-related code) fall into one of these buckets:

• Inputs: Data used to train and test our algorithms

• Variables: Mutable tensors, mostly defining the parameters of our algorithms

• Outputs: Immutable tensors storing both terminal and intermediate outputs

• Operations: Various transformations for inputs to produce the desired outputs

In our earlier example, in the sigmoid example, we can find instances of all these categories. We list the elements in Table 2.1:

TensorFlow element	Value from example client
Inputs	x
Variables	W and b
Outputs	h
Operations	tf.matmul(...), tf.nn.sigmoid(...)

The following subsections explain each of these TensorFlow elements in more detail.

Defining inputs in TensorFlow

The client...

Reusing variables with scoping

Until now, we have looked at the architecture of TensorFlow and the essentials required to implement a basic TensorFlow client. However, there is much more to TensorFlow than this. As we already saw, TensorFlow behaves quite differently from a typical Python script. For example, you cannot debug TensorFlow code in real time (as you would do a simple Python script using a Python IDE), as the computations do not happen in real time in TensorFlow (unless you are using the Eager Execution method, which was only recently in TensorFlow 1.7: https://research.googleblog.com/2017/10/eager-execution-imperative-define-by.html). In other words, TensorFlow first defines the full computational graph, does all computations on a device, and finally fetches results. Consequently, it can be quite tedious and painful to debug a TensorFlow client. This emphasizes the importance of attention to detail when implementing a TensorFlow client. Therefore, it is advised to adhere to...

Implementing our first neural network

Great! Now that you've learned the architecture, basics, and scoping mechanism of TensorFlow, it's high time that we move on and implement something moderately complex. Let's implement a neural network. Precisely, we will implement a fully connected neural network model that we discussed in Chapter 1, Introduction to Natural Language Processing.

One of the stepping stones to the introduction of neural networks is to implement a neural network that is able to classify digits. For this task, we will be using the famous MNIST dataset made available at http://yann.lecun.com/exdb/mnist/. You might feel a bit skeptical regarding our using a computer vision task rather than a NLP task. However, vision tasks can be implemented with less preprocessing and are easy to understand.

As this is our first encounter with neural networks, we will walk through the main parts of the example. However, note that I will only walk through the crucial bits of the exercise. To...

Summary

In this chapter, you took your first steps to solving NLP tasks by understanding the primary underlying platform (TensorFlow) on which we will be implementing our algorithms. First, we discussed the underlying details of TensorFlow architecture. Next, we discussed the essential ingredients of a meaningful TensorFlow client. Then we discussed a general coding practice widely used in TensorFlow known as scoping. Later, we brought all these elements together to implement a neural network to classify an MNIST dataset.

Specifically, we discussed the TensorFlow architecture lining up the explanation with an example TensorFlow client. In the TensorFlow client, we defined the TensorFlow graph. Then, when we created a session, it looked at the graph, created a GraphDef object representing the graph, and sent it to the distributed master. The distributed master looked at the graph, decided which components to use for the relevant computation, and divided it into several subgraphs to make the...

The rest of the chapter is locked

Reusing variables with scoping

Until now, we have looked at the architecture of TensorFlow and the essentials required to implement a basic TensorFlow client. However, there is much more to TensorFlow than this. As we already saw, TensorFlow behaves quite differently from a typical Python script. For example, you cannot debug TensorFlow code in real time (as you would do a simple Python script using a Python IDE), as the computations do not happen in real time in TensorFlow (unless you are using the Eager Execution method, which was only recently in TensorFlow 1.7: https://research.googleblog.com/2017/10/eager-execution-imperative-define-by.html). In other words, TensorFlow first defines the full computational graph, does all computations on a device, and finally fetches results. Consequently, it can be quite tedious and painful to debug a TensorFlow client. This emphasizes the importance of attention to detail when implementing a TensorFlow client. Therefore, it is advised to adhere to...

Implementing our first neural network

Great! Now that you've learned the architecture, basics, and scoping mechanism of TensorFlow, it's high time that we move on and implement something moderately complex. Let's implement a neural network. Precisely, we will implement a fully connected neural network model that we discussed in Chapter 1, Introduction to Natural Language Processing.

One of the stepping stones to the introduction of neural networks is to implement a neural network that is able to classify digits. For this task, we will be using the famous MNIST dataset made available at http://yann.lecun.com/exdb/mnist/. You might feel a bit skeptical regarding our using a computer vision task rather than a NLP task. However, vision tasks can be implemented with less preprocessing and are easy to understand.

As this is our first encounter with neural networks, we will walk through the main parts of the example. However, note that I will only walk through the crucial bits of the exercise. To...

Summary

In this chapter, you took your first steps to solving NLP tasks by understanding the primary underlying platform (TensorFlow) on which we will be implementing our algorithms. First, we discussed the underlying details of TensorFlow architecture. Next, we discussed the essential ingredients of a meaningful TensorFlow client. Then we discussed a general coding practice widely used in TensorFlow known as scoping. Later, we brought all these elements together to implement a neural network to classify an MNIST dataset.

Specifically, we discussed the TensorFlow architecture lining up the explanation with an example TensorFlow client. In the TensorFlow client, we defined the TensorFlow graph. Then, when we created a session, it looked at the graph, created a GraphDef object representing the graph, and sent it to the distributed master. The distributed master looked at the graph, decided which components to use for the relevant computation, and divided it into several subgraphs to make the...