ANN Technologies Using TensorFlow 2

In this chapter, we will discuss and exemplify those parts of TensorFlow 2 that are needed for the construction, training, and evaluation of artificial neural networks and their utilization purposes for inference. Initially, we will not present a complete application. Instead, we will focus on individual concepts and techniques before putting them all together and presenting full models in subsequent chapters.

In this chapter, we will cover the following topics:

• Presenting data to an Artificial Neural Network (ANN)
• ANN layers
• Gradient calculations for gradient descent algorithms
• Loss functions

Presenting data to an ANN

The canonical way to present data to a TensorFlow ANN, as recommended by Google, is via a data pipeline composed of a tf.data.Dataset object and a tf.data.Iterator method. A tf.data.Dataset object consists of a sequence of elements in which each element contains one or more tensor objects. The tf.data.Iterator is a method used to loop over a dataset so that successive individual elements in it may be accessed.

We will look at two important ways of constructing a data pipeline, firstly, from in-memory NumPy arrays, and, secondly, from Comma-Separated Value (CSV) files. We will also look at a binary TFRecord format.

Using NumPy arrays with datasets

Let's look at some straightforward examples first...

One-hot encoding

One-hot encoding (OHE) is where a tensor is constructed from the data labels with a 1 in each of the elements corresponding to a label's value, and 0 everywhere else; that is, one of the bits in the tensor is hot (1).

OHE example 1

In this example, we are converting a decimal value of 5 to a one-hot encoded value of 0000100000 using the tf.one_hot() method:

y = 5
y_train_ohe = tf.one_hot(y, depth=10).numpy() 
print(y, "is ",y_train_ohe,"when one-hot encoded with a depth of 10")
# 5 is 00000100000 when one-hot encoded with a depth of 10

OHE example 2

...

Layers

The fundamental data structure used by ANNs is the layer, and many interconnected layers make up a complete ANN. A layer can be envisaged as an array of neurons, although the use of the word neuron can be misleading, since there is only a marginal correspondence between human brain neurons and the artificial neurons that make up a layer. Bearing that in mind, we will use the term neuron in what follows. As with any computer processing unit, a neuron is characterized by its inputs and its outputs. In general, a neuron has many inputs and one output value. Each input connection carries a weight, w_i.

The following diagram shows a neuron. It is important to note that the activation function, f, is non-linear for anything other than trivial ANNs. A general neuron in the network receives inputs from other neurons and each of these carries a weight, w_i, as shown, and the network...

Activation functions

It is important to note that neural nets have non-linear activation functions, that is, the functions that are applied to the sum of the weighted inputs to a neuron. Linear activation units are not able to map input layers to output layers in all but trivial neural net models.

There are a number of activation functions in common use, including sigmoid, tanh, ReLU, and leaky ReLU. A good summary, together with diagrams of these functions, may be found here: https://towardsdatascience.com/activation-functions-neural-networks-1cbd9f8d91d6.

Creating the model

There are four methods for creating our ANN model using Keras:

• Method 1: Arguments passed to tf.keras.Sequential
• Method 2: Use of the .add method of tf.keras.Sequential
• Method 3: Use of the Keras functional API
• Method 4: By subclassing the tf.keras.Model object

Please refer to Chapter 2, Keras, a High-Level API for TensorFlow 2, for details regarding these four methods.

Gradient calculations for gradient descent algorithms

One of TenorFlow's great strengths is its ability to automatically compute gradients for use in gradient descent algorithms, which, of course, are a vital part of most machine learning models. TensorFlow offers a number of methods for gradient calculations.

There are four ways to automatically compute gradients when eager execution is enabled (they also work in graph mode):

1. tf.GradientTape: Context records computations so that you can call tf.gradient() to get the gradients of any tensor computed while recording with respect to any trainable variable
2. tfe.gradients_function(): Takes a function (say f()) and returns a gradient function (say fg()) that can compute the gradients of the outputs of f() with respect to the parameters of f() or a subset of them
3. tfe.implicit_gradients(): This is very similar, but fg() computes...

Loss functions

A loss function (that is, an error measurement) is a necessary part of the training of an ANN. It is a measure of the extent to which the calculated output of a network during training differs from its required output. By differentiating the loss function, we can find a quantity with which to adjust the weights of the connections between the layers so as to make the calculated output of the ANN more closely match the required output.

The simplest loss function is the mean squared error:

,

Here, y is the actual label value, and is the predicted label value.

Of particular note is the categorical cross-entropy loss function, which is given by the following equation:

This loss function is used when only one class is correct out of all the possible ones and so is used when the softmax function is used as the output of the final layer of an ANN.

Note that both of these...

Summary

In this chapter, we looked at a number of technologies that support the creation and use of neural networks.

We covered data presentation to an ANN, layers of an ANN, creating the model, gradient calculations for gradient descent algorithms, loss functions, and saving and restoring models. These topics are important precursors to the concepts and techniques that we will encounter in subsequent chapters when we develop neural network models.

Indeed, in the next chapter, we will start to use TensorFlow in earnest by exploring a number of supervised learning scenarios, including linear regression, logistic regression, and k-nearest neighbors.