Understanding TensorFlow 2

In this chapter, you will get an in-depth understanding of TensorFlow. This is an open source distributed numerical computation framework, and it will be the main platform on which we will be implementing all our exercises. This chapter covers the following topics:

• What is TensorFlow?
• The building blocks of TensorFlow (for example, variables and operations)
• Using Keras for building models
• Implementing our first neural network

We will get started with TensorFlow by defining a simple calculation and trying to compute it using TensorFlow. After we complete this, we will investigate how TensorFlow executes this computation. This will help us to understand how the framework creates a computational graph to compute the outputs and execute this graph to obtain the desired outputs. Then we will dive into the details of how TensorFlow architecture operates by looking at how TensorFlow executes things, with the help of an analogy...

What is TensorFlow?

In Chapter 1, Introduction to Natural Language Processing, we briefly discussed what TensorFlow is. Now let’s take a closer look at it. TensorFlow is an open source, distributed numerical computation framework released by Google that is mainly intended to alleviate the painful details of implementing a neural network (for example, computing derivatives of the weights of the neural network). TensorFlow takes this a step further by providing efficient implementations of such numerical computations using Compute Unified Device Architecture (CUDA), which is a parallel computational platform introduced by NVIDIA (for more information on CUDA, visit https://blogs.nvidia.com/blog/2012/09/10/what-is-cuda-2/). The Application Programming Interface (API) of TensorFlow at https://www.tensorflow.org/api_docs/python/tf/all_symbols shows that TensorFlow provides thousands of operations that make our lives easier.

TensorFlow was not developed overnight. This is a result...

Inputs, variables, outputs, and operations

Now we are returning from our journey into TensorFlow 1 and stepping back to TensorFlow 2. Let’s proceed to the most common elements that comprise a TensorFlow 2 program. If you read any of the millions of TensorFlow clients available on the internet, the TensorFlow-related code all falls into one of these buckets:

• Inputs: Data used to train and test our algorithms
• Variables: Mutable tensors, mostly defining the parameters of our algorithms
• Outputs: Immutable tensors storing both terminal and intermediate outputs
• Operations: Various transformations for inputs to produce the desired outputs

In our earlier sigmoid example, we can find instances of all these categories. We list the respective TensorFlow elements and the notation used in the sigmoid example in Table 2.1:

TensorFlow element

Value from example client ...

Keras: The model building API of TensorFlow

Keras was developed as a separate library that provides high-level building blocks to build models conveniently. It was initially platform-agnostic and supported many softwares (for example, TensorFlow and Theano).

However, TensorFlow acquired Keras and now is an integral part of TensorFlow for building models effortlessly.

Keras’s primary focus is model building. For that, Keras provides several different APIs with varying degrees of flexibility and complexity. Choosing the right API for the job will require sound knowledge of the limitations of each API as well as experience. The APIs provided by Keras are:

• Sequential API – The most easy-to-use API. In this API, you simply stack layers on top of each other to create a model.
• Functional API – The functional API provides more flexibility by allowing you to define custom models that can have multiple input layers/multiple output layers.
...

Implementing our first neural network

Great! Now that you’ve learned the architecture and foundations of TensorFlow, it’s high time that we move on and implement something slightly more complex. Let’s implement a neural network. Specifically, we will implement a fully connected neural network model (FCNN), which we discussed in Chapter 1, Introduction to Natural Language Processing.

One of the stepping stones to the introduction of neural networks is to implement a neural network that is able to classify digits. For this task, we will be using the famous MNIST dataset made available at http://yann.lecun.com/exdb/mnist/.

You might feel a bit skeptical regarding our using a computer vision task rather than an NLP task. However, vision tasks can be implemented with less preprocessing and are easy to understand.

As this is our first encounter with neural networks, we will see how to implement this model using Keras. Keras is the high-level submodule that...

Summary

In this chapter, you took your first steps to solving NLP tasks by understanding the primary underlying platform (TensorFlow) on which we will be implementing our algorithms. First, we discussed the underlying details of TensorFlow architecture. Next, we discussed the essential ingredients of a meaningful TensorFlow program. We got to know some new features in TensorFlow 2, such as the AutoGraph feature, in depth. We then discussed more exciting elements in TensorFlow such as data pipelines and various TensorFlow operations.

Specifically, we discussed the TensorFlow architecture by lining up the explanation with an example TensorFlow program; the sigmoid example. In this TensorFlow program, we used the AutoGraph feature to generate a TensorFlow graph; that is, using the tf.function() decorator over the function that performs the TensorFlow operations. Then, a GraphDef object was created representing the graph and sent to the distributed master. The distributed master...