TensorFlow Machine Learning Projects

Tensors are the basic components in TensorFlow. A tensor is a multidimensional collection of data elements. It is generally identified by shape, type, and rank. Rank refers to the number of dimensions of a tensor, while shape refers to the size of each dimension. You may have seen several examples of tensors before, such as in a zero-dimensional collection (also known as a scalar), a one-dimensional collection (also known as a vector), and a two-dimensional collection (also known as a matrix).

A scalar value is a tensor of rank 0 and shape []. A vector, or a one-dimensional array, is a tensor of rank 1 and shape [number_of_columns] or [number_of_rows]. A matrix, or a two-dimensional array, is a tensor of rank 2 and shape [number_of_rows, number_of_columns]. A three-dimensional array is a tensor of rank 3. In the same way, an n-dimensional array is a tensor of rank n.

A tensor can store data of one type in all of its dimensions, and the data type of a tensor is the same as the data type of its elements.

The following are the most commonly used data types in TensorFlow:

Tensors can be created in the following ways:

Let's explore different ways of creating Tensors.

The constant valued tensors are created using the tf.constant() function, and has the following definition:

tf.constant(
  value,
  dtype=None,
  shape=None,
  name='const_name',
  verify_shape=False
  )

Let's create some constants with the following code:

const1=tf.constant(34,name='x1')
const2=tf.constant(59.0,name='y1')
const3=tf.constant(32.0,dtype=tf.float16,name='z1')

Let's take a look at the preceding code in detail:

Let's print the constants const1, const2, and const3:

print('const1 (x): ',const1)
print('const2 (y): ',const2)
print('const3 (z): ',const3)

When we print these constants, we get the following output:

const1 (x):  Tensor("x:0", shape=(), dtype=int32)
const2 (y):  Tensor("y:0", shape=(), dtype=float32)
const3 (z):  Tensor("z:0", shape=(), dtype=float16)

To print the values of these constants, we can execute them in a TensorFlow session with the tfs.run() command:

print('run([const1,const2,c3]) : ',tfs.run([const1,const2,const3]))

We will see the following output:

run([const1,const2,const3]) : [34, 59.0, 32.0]

The TensorFlow library contains several built-in operations that can be applied on tensors. An operation node can be defined by passing input values and saving the output in another tensor. To understand this better, let's define two operations, op1 and op2:

op1 = tf.add(const2, const3)
op2 = tf.multiply(const2, const3)

Let's print op1 and op2:

print('op1 : ', op1)
print('op2 : ', op2)

The output is as follows, and shows that op1 and op2 are defined as tensors:

op1 :  Tensor("Add:0", shape=(), dtype=float32)
op2 :  Tensor("Mul:0", shape=(), dtype=float32)

To print the output from executing these operations, the op1 and op2 tensors have to be executed in a TensorFlow session:

print('run(op1) : ', tfs.run(op1))
print('run(op2) : ', tfs.run(op2))

The output is as follows:

run(op1) :  91.0
run(op2) :  1888.0

Some of the built-in operations of TensorFlow include arithmetic operations, math functions, and complex number operations.

While constants store the value at the time of defining the tensor, placeholders allow you to create empty tensors so that the values can be provided at runtime. The TensorFlow library provides the tf.placeholder() function with the following signature to create placeholders:

tf.placeholder(
  dtype,
  shape=None,
  name=None
  )

As an example, let's create two placeholders and print them:

p1 = tf.placeholder(tf.float32)
p2 = tf.placeholder(tf.float32)
print('p1 : ', p1)
print('p2 : ', p2)

The following output shows that each placeholder has been created as a tensor:

p1 :  Tensor("Placeholder:0", dtype=float32)
p2 :  Tensor("Placeholder_1:0", dtype=float32)

Let's define an operation using these placeholders:

mult_op = p1 * p2

In TensorFlow, shorthand symbols can be used for various operations. In the preceding code, p1 * p2 is shorthand for tf.multiply(p1,p2):

print('run(mult_op,{p1:13.4, p2:61.7}) : ',tfs.run(mult_op,{p1:13.4, p2:61.7}))

The preceding command runs mult_op in the TensorFlow session and feeds the values dictionary (the second argument to the run() operation) with the values for p1 and p2.

The output is as follows: 

run(mult_op,{p1:13.4, p2:61.7}) :  826.77997

We can also specify the values dictionary by using the feed_dict parameter in the run() operation:

feed_dict={p1: 15.4, p2: 19.5}
print('run(mult_op,feed_dict = {p1:15.4, p2:19.5}) : ',
      tfs.run(mult_op, feed_dict=feed_dict))

The output is as follows:

run(mult_op,feed_dict = {p1:15.4, p2:19.5}) :  300.3

Let's look at one final example, which is of a vector being fed to the same operation:

feed_dict={p1: [2.0, 3.0, 4.0], p2: [3.0, 4.0, 5.0]}
print('run(mult_op,feed_dict={p1:[2.0,3.0,4.0], p2:[3.0,4.0,5.0]}):',
      tfs.run(mult_op, feed_dict=feed_dict))

The output is as follows:

run(mult_op,feed_dict={p1:[2.0,3.0,4.0],p2:[3.0,4.0,5.0]}):[  6.  12.  20.]

The elements of the two input vectors are multiplied in an element-wise fashion.

Tensors can be created from Python objects such as lists, NumPy arrays, and pandas DataFrames. To create tensors from Python objects, use the tf.convert_to_tensor() function with the following definition:

tf.convert_to_tensor(
  value,
  dtype=None,
  name=None,
  preferred_dtype=None
  )

Let's practice doing this by creating some tensors and printing their definitions and values:

tf_t=tf.convert_to_tensor(5.0,dtype=tf.float64)

print('tf_t : ',tf_t)
print('run(tf_t) : ',tfs.run(tf_t))

The output is as follows: 

tf_t : Tensor("Const_1:0", shape=(), dtype=float64)
run(tf_t) : 5.0

a1dim = np.array([1,2,3,4,5.99])
print("a1dim Shape : ",a1dim.shape)

tf_t=tf.convert_to_tensor(a1dim,dtype=tf.float64)

print('tf_t : ',tf_t)
print('tf_t[0] : ',tf_t[0])
print('tf_t[0] : ',tf_t[2])
print('run(tf_t) : \n',tfs.run(tf_t))

The output is as follows:

a1dim Shape :  (5,)
tf_t :  Tensor("Const_2:0", shape=(5,), dtype=float64)
tf_t[0] :  Tensor("strided_slice:0", shape=(), dtype=float64)
tf_t[0] :  Tensor("strided_slice_1:0", shape=(), dtype=float64)
run(tf_t) : 
 [ 1.    2.    3.    4.    5.99]

a2dim = np.array([(1,2,3,4,5.99),
                  (2,3,4,5,6.99),
                  (3,4,5,6,7.99)
                 ])
print("a2dim Shape : ",a2dim.shape)

tf_t=tf.convert_to_tensor(a2dim,dtype=tf.float64)

print('tf_t : ',tf_t)
print('tf_t[0][0] : ',tf_t[0][0])
print('tf_t[1][2] : ',tf_t[1][2])
print('run(tf_t) : \n',tfs.run(tf_t))

The output is as follows:

a2dim Shape :  (3, 5)
tf_t :  Tensor("Const_3:0", shape=(3, 5), dtype=float64)
tf_t[0][0] :  Tensor("strided_slice_3:0", shape=(), dtype=float64)
tf_t[1][2] :  Tensor("strided_slice_5:0", shape=(), dtype=float64)
run(tf_t) : 
 [[ 1.    2.    3.    4.    5.99]
  [ 2.    3.    4.    5.    6.99]
  [ 3.    4.    5.    6.    7.99]]

a3dim = np.array([[[1,2],[3,4]],
                  [[5,6],[7,8]]
                 ])
print("a3dim Shape : ",a3dim.shape)

tf_t=tf.convert_to_tensor(a3dim,dtype=tf.float64)

print('tf_t : ',tf_t)
print('tf_t[0][0][0] : ',tf_t[0][0][0])
print('tf_t[1][1][1] : ',tf_t[1][1][1])
print('run(tf_t) : \n',tfs.run(tf_t))

The output is as follows:

a3dim Shape :  (2, 2, 2)
tf_t :  Tensor("Const_4:0", shape=(2, 2, 2), dtype=float64)
tf_t[0][0][0] :  Tensor("strided_slice_8:0", shape=(), dtype=float64)
tf_t[1][1][1] :  Tensor("strided_slice_11:0", shape=(), dtype=float64)
run(tf_t) : 
 [[[ 1.  2.][ 3.  4.]]
  [[ 5.  6.][ 7.  8.]]]

In the previous sections, we learned how to define tensor objects of different types, such as constants, operations, and placeholders. The values of parameters need to be held in an updatable memory location while building and training models with TensorFlow. Such updatable memory locations for tensors are known as variables in TensorFlow.

To summarize this, TensorFlow variables are tensor objects in that their values can be modified during the execution of the program.

Although tf.Variable seems to be similar to tf.placeholder, they have certain differences. These are listed in the following table:

In TensorFlow, a variable can be created with the API function tf.Variable(). Let's look at an example of using placeholders and variables and create the following model in TensorFlow:

w = tf.Variable([.3], tf.float32)
b = tf.Variable([-.3], tf.float32)

x = tf.placeholder(tf.float32)
y = w * x + b

print("w:",w)
print("x:",x)
print("b:",b)
print("y:",y)

The output depicts the type of nodes as Variable, Placeholder, or operation node, as follows:

w: <tf.Variable 'Variable:0' shape=(1,) dtype=float32_ref>
x: Tensor("Placeholder_2:0", dtype=float32)
b: <tf.Variable 'Variable_1:0' shape=(1,) dtype=float32_ref>
y: Tensor("add:0", dtype=float32)

The preceding output indicates that x is a Placeholder tensor, y is an operation tensor, and that w and b are variables with a shape of (1,) and a data type of float32.

The variables in a TensorFlow session have to be initialized before they can be used. We can either initialize a single variable by running its initializer operation or we can initialize all or a group of variables.

For example, to initialize the w variable, we can use the following code:

tfs.run(w.initializer)

TensorFlow provides a convenient function that can initialize all of the variables:

tfs.run(tf.global_variables_initializer())

The global convenience function for initializing these variables can be executed in an alternative way. Instead of executing inside the run() function of a session object, the run function of the object returned by the initializer function itself can be executed:

tf.global_variables_initializer().run()

After the variables have been initialized, execute the model to get the output for the input values of x = [1,2,3,4]:

print('run(y,{x:[1,2,3,4]}) : ',tfs.run(y,{x:[1,2,3,4]}))

The output is as follows:

run(y,{x:[1,2,3,4]}) :  [ 0.          0.30000001  0.60000002  0.90000004]

TensorFlow provides various functions to generate tensors with pre-populated values. The generated values from these functions can be stored in a constant or variable tensor. Such generated values can also be provided to the tensor constructor at the time of initialization.

As an example, let's generate a 1-D tensor that's been pre-populated with 100 zeros:

a=tf.zeros((100,))
print(tfs.run(a))

Some of the TensorFlow library functions that populate these tensors with different values at the time of their definition are listed as follows:

If a variable is defined with a name that has already been used for another variable, then an exception is thrown by TensorFlow. Thetf.get_variable() function makes it convenient and safe to create a variable in place of using thetf.Variable() function. The tf.get_variable() function returns a variable that has been defined with a given name. If the variable with the given name does not exist, then it will create the variable with the specified initializer and shape.

Consider the following example:

w = tf.get_variable(name='w',shape=[1],dtype=tf.float32,initializer=[.3])
b = tf.get_variable(name='b',shape=[1],dtype=tf.float32,initializer=[-.3])

The initializer can either be a list of values or another tensor. An initializer can also be one of the built-in initializers. Some of these are as follows:

The tf.get_variable() function only returns the global variables when the code is run across multiple machines in distributed TensorFlow. The local variables can be retrieved by using the tf.get_local_variable() function.

Now that we have learned how to define tensors, constants, operations, placeholders, and variables, let's learn about the next level of abstraction in TensorFlow that combines these basic elements to form a basic unit of computation: the computation graph.

The nodes in a computation graph are executed in their order of dependency. If node x depends on node y, then x is executed before y when the execution of y is requested. A node is only executed if either the node itself or another node depending on it is invoked for execution. This execution philosophy is known as lazy loading. As the name implies, the node objects are not instantiated and initialized until they are actually required.

Often, it is necessary to control the order of the execution of the nodes in a computation graph. This can be done with the tf.Graph.control_dependencies() function. For example, if the graph has the nodes l,m,n, and o, and we want to execute n and o before l and m, then we would use the following code:

with graph_variable.control_dependencies([n,o]):
  # other statements here

This makes sure that any node in the preceding with block is executed after nodes n and o have been executed.

A graph can be partitioned into several parts, and each part can be placed and executed on different devices, such as a CPU or GPU. All of the devices that are available for graph execution can be listed with the following command:

from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())

The output is listed as follows (the output for your machine will be different because this will depend on the available compute devices in your specific system):

[name: "/device:CPU:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 12900903776306102093
, name: "/device:GPU:0"
device_type: "GPU"
memory_limit: 611319808
locality {
  bus_id: 1
}
incarnation: 2202031001192109390
physical_device_desc: "device: 0, name: Quadro P5000, pci bus id: 0000:01:00.0, compute capability: 6.1"
]

The devices in TensorFlow are identified with the string /device:<device_type>:<device_idx>. In the last output, CPU and GPU denote the device type, and 0 denotes the device index.

One thing to note about the last output is that it shows only one CPU, whereas our computer has 8 CPUs. The reason for this is that TensorFlow implicitly distributes the code across the CPU units and thus, by default, CPU:0 denotes all of the CPUs available to TensorFlow. When TensorFlow starts executing graphs, it runs the independent paths within each graph in a separate thread, with each thread running on a separate CPU. We can restrict the number of threads used for this purpose by changing the number of inter_op_parallelism_threads. Similarly, if, within an independent path, an operation is capable of running on multiple threads, TensorFlow will launch that specific operation on multiple threads. The number of threads in this pool can be changed by setting the number of intra_op_parallelism_threads.

tf.reset_default_graph()

# Define model parameters
w = tf.Variable([.3], tf.float32)
b = tf.Variable([-.3], tf.float32)
# Define model input and output
x = tf.placeholder(tf.float32)
y = w * x + b

config = tf.ConfigProto()
config.log_device_placement=True

with tf.Session(config=config) as tfs:
# initialize and print the variable y
tfs.run(global_variables_initializer())
   print('output',tfs.run(y,{x:[1,2,3,4]}))

b: (VariableV2): /job:localhost/replica:0/task:0/device:GPU:0
b/read: (Identity): /job:localhost/replica:0/task:0/device:GPU:0
b/Assign: (Assign): /job:localhost/replica:0/task:0/device:GPU:0
w: (VariableV2): /job:localhost/replica:0/task:0/device:GPU:0
w/read: (Identity): /job:localhost/replica:0/task:0/device:GPU:0
mul: (Mul): /job:localhost/replica:0/task:0/device:GPU:0
add: (Add): /job:localhost/replica:0/task:0/device:GPU:0
w/Assign: (Assign): /job:localhost/replica:0/task:0/device:GPU:0
init: (NoOp): /job:localhost/replica:0/task:0/device:GPU:0
x: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
b/initial_value: (Const): /job:localhost/replica:0/task:0/device:GPU:0
Const_1: (Const): /job:localhost/replica:0/task:0/device:GPU:0
w/initial_value: (Const): /job:localhost/replica:0/task:0/device:GPU:0
Const: (Const): /job:localhost/replica:0/task:0/device:GPU:0

tf.reset_default_graph()

with tf.device('/device:CPU:0'):
# Define model parameters
w = tf.get_variable(name='w',initializer=[.3], dtype=tf.float32)
    b = tf.get_variable(name='b',initializer=[-.3], dtype=tf.float32)
# Define model input and output
x = tf.placeholder(name='x',dtype=tf.float32)
    y = w * x + b

config = tf.ConfigProto()
config.log_device_placement=True

with tf.Session(config=config) as tfs:
# initialize and print the variable y
tfs.run(tf.global_variables_initializer())
   print('output',tfs.run(y,{x:[1,2,3,4]}))

b: (VariableV2): /job:localhost/replica:0/task:0/device:CPU:0
b/read: (Identity): /job:localhost/replica:0/task:0/device:CPU:0
b/Assign: (Assign): /job:localhost/replica:0/task:0/device:CPU:0
w: (VariableV2): /job:localhost/replica:0/task:0/device:CPU:0
w/read: (Identity): /job:localhost/replica:0/task:0/device:CPU:0
mul: (Mul): /job:localhost/replica:0/task:0/device:CPU:0
add: (Add): /job:localhost/replica:0/task:0/device:CPU:0
w/Assign: (Assign): /job:localhost/replica:0/task:0/device:CPU:0
init: (NoOp): /job:localhost/replica:0/task:0/device:CPU:0
x: (Placeholder): /job:localhost/replica:0/task:0/device:CPU:0
b/initial_value: (Const): /job:localhost/replica:0/task:0/device:CPU:0
Const_1: (Const): /job:localhost/replica:0/task:0/device:CPU:0
w/initial_value: (Const): /job:localhost/replica:0/task:0/device:CPU:0
Const: (Const): /job:localhost/replica:0/task:0/device:CPU:0

If the graph was previously run, 
    then the node is left on the device where it was placed earlier
Else If the tf.device() block is used,
    then the node is placed on the specified device
Else If the GPU is present
    then the node is placed on the first available GPU
Else If the GPU is not present
    then the node is placed on the CPU

config.allow_soft_placement = True

os.environ['CUDA_VISIBLE_DEVICES']='0'

config.gpu_options.per_process_gpu_memory_fraction = 0.5

config.gpu_options.allow_growth = True

We can create our own graphs, which are separate from the default graph, and execute them in a session. However, creating and executing multiple graphs is not recommended, because of the following disadvantages:

Hence, the recommended approach is to have multiple subgraphs in a single graph. In case we wish to use our own graph instead of the default graph, we can do so with the tf.graph() command. In the following example, we create our own graph, g, and execute it as the default graph:

g = tf.Graph()
output = 0

# Assume Linear Model y = w * x + b

with g.as_default():
 # Define model parameters
 w = tf.Variable([.3], tf.float32)
 b = tf.Variable([-.3], tf.float32)
 # Define model input and output
 x = tf.placeholder(tf.float32)
 y = w * x + b

with tf.Session(graph=g) as tfs:
 # initialize and print the variable y
 tf.global_variables_initializer().run()
 output = tfs.run(y,{x:[1,2,3,4]})

print('output : ',output)

Now, let's put this learning into practice and implement the classification of handwritten digital images with TensorFlow.

Machine learning refers to the application of algorithms to make computers learn from data. The models that are learned by computers are used to make predictions and forecasts. Machine learning has been successfully applied in a variety of areas, such as natural language processing, self-driving vehicles, image and speech recognition, chatbots, and computer vision.

Machine learning algorithms are broadly categorized into three types:

These machine learning problems are abstracted to the following equation in one form or another:

Here, y represents the target and x represents the feature. If x is a collection of features, it is also called a feature vector and denoted with X. The model is the function f that maps features to targets. Once the computer learns f, it can use the new values of x to predict the values of y.

The preceding simple equation can be rewritten in the context of linear models for machine learning as follows:

Here, w is known as the weight and b is known as the bias. Thus, the machine learning problem now can be stated as a problem of finding w and b from the current values of X so that the equation can now be used to predict the values of y. 

Regression analysis or regression modeling refers to the methods and techniques used to estimate relationships among variables. The variables that are used as input for regression models are called independent variables, predictors, or features, and the output variables from regression models are called dependent variables or targets. Regression models are defined as follows:

Where Y is the target variable, X is a vector of features, and β is a vector of parameters (w,b in the preceding equation).

Classification is one of the classical problems in machine learning. Data under consideration could belong to one class or another, for example, if the images provided are data, they could be pictures of cats or dogs. Thus, the classes, in this case, are cats and dogs. Classification means identifying the label or class of the objects under consideration. Classification falls under the umbrella of supervised machine learning. In classification problems, a training dataset is provided that has features or inputs and their corresponding outputs or labels. Using this training dataset, a model is trained; in other words, the parameters of the model are computed. The trained model is then used on new data to find its correct labels.

Classification problems can be of two types: binary class or multiclass. Binary class means that the data is to be classified into two distinct and discrete labels; for example, the patient has cancer or the patient does not have cancer, and the images are of cats or dogs and so on. Multiclass means that the data is to be classified among multiple classes, for example, an email classification problem will divide emails into social media emails, work-related emails, personal emails, family-related emails, spam emails, shopping offer emails, and so on. Another example would be of pictures of digits; each picture could be labeled between 0 and 9, depending on what digit the picture represents. In this chapter, we will look at examples of both kinds of classification.

The most popular method for classification is logistic regression. Logistic regression is a probabilistic and linear classifier. The probability that the vector of input features belongs to a specific class can be described mathematically by the following equation: 

In the preceding equation, the following applies:

The ϕ(z) function represents the probability that x belongs to class i when w and b are given. Thus, the model has to be trained to maximize the value of this probability.

For binary classification, the model function ϕ(z) is defined as the sigmoid function, which can be described as follows:

The sigmoid function transforms the y value to be between the range [0,1]. Thus, the value of y=ϕ(z) can be used to predict the class: if y > 0.5, then the object belongs to 1, otherwise the object belongs to 0.

The model training means to search for the parameters that minimize the loss function, which can either be the sum of squared errors or the sum of mean squared errors. For logistic regression, the likelihood is maximized as follows:

However, as it is easier to maximize the log-likelihood, we use the log-likelihood (l(w)) as the cost function. The loss function (J(w)) is written as -l(w), and can be minimized by using optimization algorithms such as gradient descent.

The loss function for binary logistic regression is written mathematically as follows:

Here, ϕ(z) is the sigmoid function.

When more than two classes are involved, logistic regression is knownas multinomial logistic regression. In multinomial logistic regression, instead of sigmoid, use the softmax function, which can be described mathematically as follows:

The softmax function produces the probabilities for each class so that the probabilities vector adds up to 1. At the time of inference, the class with the highest softmax value becomes the output or predicted class. The loss function, as we discussed earlier, is the negative log-likelihood function, -l(w), that can be minimized by the optimizers, such as gradient descent.

The loss function for multinomial logistic regression is written formally as follows:

Here, ϕ(z) is the softmax function.

We will implement this loss function in the next section. In the following section, we will dig into our example for multiclass classification with logistic regression in TensorFlow.