TensorFlow Graph Architecture

The most concise and complete explanation of what TensorFlow is can be found on the project home page (https://www.tensorflow.org/) and it highlights every important part of the library. TensorFlow is an open source software library for high-performance numerical computation. Its flexible architecture allows easy deployment of computation across a variety of platforms (CPUs, GPUs, and TPUs), from desktops to clusters of servers, to mobile and edge devices. Originally developed by researchers and engineers from the Google Brain team within Google's AI organization, it comes with strong support for machine learning and deep learning, and the flexible numerical computation core is used across many other scientific domains.

TensorFlow's strengths and most important features can be summarized in the following three points:

• High-performance numerical...

Environment setup

In order to understand the structure of TensorFlow, all the examples presented in this chapter will use the latest TensorFlow 1.x release: 1.15; however, we will also set up everything needed to run TensorFlow 2.0 since we are going to use it in the next chapter, Chapter 4, TensorFlow 2.0 Architecture.

All the examples presented in this book specify the version of TensorFlow to use when running it. Being a library, we can just install it specifying the version we need. Of course, having two different versions of the same library installed on one system would be a mistake. In order to be able to switch between versions, we are going to use two different Python virtual environments.

An explanation of what a virtual environment (virtualenv) is and why it perfectly fits our needs follows here, from the official introduction to virtual environments (https://docs.Python...

Dataflow graphs

In order to be a highly efficient, flexible, and production-ready library, TensorFlow uses dataflow graphs to represent computation in terms of the relationships between individual operations. Dataflow is a programming model widely used in parallel computing and, in a dataflow graph, the nodes represent units of computation while the edges represent the data consumed or produced by a computation unit.

As seen in the previous chapter, Chapter 2, Neural Networks and Deep Learning, representing computation using graphs comes with the advantage of being able to run the forward and backward passes required to train a parametric machine learning model via gradient descent, applying the chain rule to compute the gradient as a local process to every node; however, this is not the only advantage of using graphs.

Reducing the abstraction level and thinking about the implementation...

Model definition and training

Disclaimer: the layer module has been completely removed in TensorFlow 2.0, and the layer definition using tf.keras.layers is the new standard; however, an overview of tf.layers is still worth reading because it shows how reasoning layer by layer to define deep models is the natural way to proceed and it also gives us an idea of the reasons behind the migration from tf.layers to tf.keras.layers.

Defining models with tf.layers

As shown in the previous section, TensorFlow provides all the primitive features to define a neural network layer: the user should take care when defining the variables, the operation nodes, the activation functions, and the logging, and define a proper interface to handle...

Interacting with the graph using Python

Python is the language of choice to train a TensorFlow model; however, after defining a computational graph in Python, there are no constraints regarding using it with another language to execute the learning operations defined.

Always keep in mind that we use Python to define a graph and this definition can be exported in a portable and language-agnostic representation (Protobuf)—this representation can then be used in any other language to create a concrete graph and using it within a session.

The TensorFlow Python API is complete and easy to use. Therefore, we can extend the previous example to measure the accuracy (defining the accuracy measurement operation in the graph) and use this metric to perform model selection.

Selecting the best model means storing the model parameters at the end of each epoch and moving the parameters...

Summary

In this chapter, we analyzed how TensorFlow works under the hood—the separation between the graph definition phase and its execution within a session, how to use the Python API to interact with a graph, and how to define a model and measure the metrics during training.

It's worth noting that this chapter analyzed how TensorFlow works in its static graph version, which is no longer the default in TensorFlow 2.0; however, the graph is still present and even when used in eager mode, every API call produces operations that can be executed inside a graph to speed up execution. As will be shown in the next chapter, TensorFlow 2.0 still allows models to be defined in static graph mode, especially when defining models using the Estimator API.

Having knowledge of graph representation is of fundamental importance, and having at least an intuitive idea about the advantages...

Exercises

1. Why is it possible to assess that the model suffers from overfitting only by looking at the graph?
2. Extend the baseline example to place the matrix multiplication operation on a remote device at IP 192.168.1.12; visualize the result on TensorBoard.
3. Is it necessary to have a remote device to place an operation on?
4. Extend the CNN architecture defined in the define_cnn method: add a batch normalization layer (from tf.layers) between the output of the convolutional layer and its activation function.
5. Try to train the model with the extended CNN architecture: the batch normalization layer adds two update operations that must be executed before running the training operation. Become familiar with the tf.control_dependencies method to force the execution of the operations contained inside the collection tf.GraphKeys.UPDATE_OPS, to be executed before the train operation (look...