Implementing a Deep Learning Model Using Keras

In chapter, Chapter 2, Deep Dive into Deep Neural Networks, we learned about deep learning in detail, which means we have a solid foundation in this area. We are also closer to implementing computer vision solutions for self-driving cars. In this chapter, we will learn about the deep learning API Keras. This will help us with the implementation of deep learning models. We will also examine a deep learning implementation using the Auto-Mpg dataset. We'll start by understanding what Keras is, and then implement our first deep learning model.

In this chapter, we will cover the following topics:

• Starting work with Keras
• Keras for deep learning 
• Building your first deep learning model

Let's get started!

Starting work with Keras

What is Keras? Keras is a Python-based deep learning framework that is actually the high-level API of TensorFlow. Keras can run on top of Theano, TensorFlow, or Microsoft Cognitive Toolkit (CNTK). Since it can run on any of these frameworks, Keras is amazingly simple and popular to work with; building models is as simple as stacking layers. We can make models, define layers, or set up multiple input-output models that are handled using the Keras high-level API.

Initially, Keras was developed as part of the research effort associated with the Open-Ended Neuro-Electronic Intelligent Robot Operating System (ONEIROS) project. Click on the following link to find out more: http://keras.io/.

Keras has attracted lots of attention in recent years as it is open source and, most importantly, is being actively developed by contributors from all over the world. The documentation related to Keras is endless. Yes, we may have understood the documentation...

Advantages of Keras 

Keras follows the best practices associated with reducing cognitive load. It offers simple and consistent APIs and affords us the freedom to design our own architecture.

Keras provides clear feedback on user error, which minimizes the number of user actions required. It provides high flexibility as it integrates with lower-level deep learning languages such as TensorFlow. You can implement anything that was built in the base language.

Keras also supports various programming languages. We can develop Keras in Python, as well as R. We can also run the code with TensorFlow, CNTK, Theano, and MXNet, which can be run on the CPU, TPU, and GPU as well. The best part is that it supports both NVIDIA and AMD GPUs. These advantages offered by Keras ensure that producing models with Keras is really simple. It can run with TensorFlow Serving, GPU acceleration (web Keras, Keras.js), Android (TF, TF Lite), iOS (Native CoreML), and Raspberry Pi.

In the next...

The working principle behind Keras

The main idea behind the Keras development is to facilitate experimentation's by fast prototyping. The is great to go from an idea to result with the least possible delay is key to good research. The structure in Keras is the Model that defines the complete graph of a network. To create a custom model for a project, we simply add more layers to the existing model.

Let's look at the model architecture in Keras in the following screenshot:

Fig 3.1: Model architecture in Keras

Keras relies on its backend for low-level operations like convolutions and tensor products. While Keras supports several backend engines, its default backend is TensorFlow, with Google as its primary supporter.

In the next section, we will learn about the sequential model and the functional model of Keras.

Building Keras models

There are two major models that Keras offers, as follows:

• The sequential model
• The functional model

We'll look at both in more detail in the following subsections.

The sequential model

The sequential model is like a linear stack of layers. It is useful for building simple models, such as the simple classification network and encoder-decoder models. It basically treats the layer as an object that is fed into the next layer.

For most problems, the sequential API lets you create layer-by-layer models. It restricts us from creating models that share layers or have multiple inputs or outputs.

Let's look at the Python code for this:

1. Let's begin by importing the key Python libraries:

In[1]: import tensorflow as tf
In[2]: from tensorflow import keras
In[3]: from tensorflow.keras import layers

2. We will define the model as a sequential model (In[4]) and then add a flatten layer. With the hidden layer, we have 120 neurons (In[6], In[7]), and the activation function is Rectified Linear Units (ReLU). In[8] is the last layer as it has 10 neurons and a softmax function; it turns logits into probabilities that sum to one:

In[4]: model = tf.keras...

The functional model

The functional model is the more widely used of the two models. The key aspects of such a model are as follows:

• Multi-input, multi-output, and arbitrary static graph topologies 
• Multi-input and multi-output models
• The complex model, which forks into two or more branches 
• Models with shared layers

The functional API allows you to create models that are much more versatile as you can easily identify models that link layers to more than just the previous and next layers. You can actually connect layers to any other layer and create your own complex layer.

The following steps are similar to the sequential model's implementation, but with a number of changes. Here, we'll import the model, work on its architecture, and then train the network:

In[1]: import tensorflow as tf
In[2]: from tensorflow import keras
In[3]: from tensorflow.keras import layers

In[4]: inputs = keras.Input(shape=(10,))
In[5]: x= layers.Dense(20, activation='relu')(x)
In[6...

Types of Keras execution

There are two types of execution in Keras:

• Deferred (symbolic) execution
• Eager (imperative) execution 

In deferred execution, we use Python to build a computation graph first, and then it's compiled so that it can be executed later. In eager execution, the Python runtime itself becomes the execution runtime for all the models. 

In the next section, we will learn about deep learning using Keras. We will also look into visual recognition challenges, due to which deep learning became popular.

Keras for deep learning

Deep learning started to gain popularity a couple of years ago when AlexNet, a convolutional neural network (CNN) designed by Alex Krizhevsky and published with Ilya Sutskever and doctoral adviser Geoffrey Hinton, also referred to as the godfather of deep learning, was created. AlexNet blew away the ImageNet Large Scale Visual Recognition Challenge on 30 September 2012. Their deep neural network was significantly better than all the other submissions. Architectures such as AlexNet have revolutionized the field of computer vision. In the following diagram, you can see the top five predictions for the visual challenge where AlexNet emerged victorious:

Fig 3.2: Visual Recognition Challenge 2012

Because deep learning requires lots of GPU computation and data, people began to take notice and implemented their own deep neural networks for different tasks, resulting in a deep learning library.

Theano was one of the first widely adopted deep learning...

Building your first deep learning model 

In this section, we will be using Keras with a TensorFlow 2.0 backend to perform our deep learning operations.

We will start with a dataset that contains details regarding the technical specifications of cars. This dataset can be downloaded from the UCI Machine Learning Repository. The data we will be working with in this chapter isn't images. Right now, our focus is on how to use Keras for general machine learning. Once we've learned about CNNs, we can expand Keras so that we can feed image data into the network. This section focuses on learning the basics of building a neural network with Keras.

In the next section, we will concentrate on how to use Keras and its general syntax.

Description of the Auto-Mpg dataset

To begin with, we will use the Auto-Mpg dataset. This dataset can be downloaded from the UCI Machine Learning Repository. The following is some information about this dataset:

• Title: Auto-Mpg Data
• Sources:
• Origin: This dataset was taken from the StatLib library, which is maintained at Carnegie Mellon University. The dataset was used in the 1983 American Statistical Association Exposition.
• Date: July 7, 1993.

•  Past usage:
• See the date in the preceding bullet point.
• Quinlan, R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann.
• Relevant information: This dataset is a slightly modified version of the dataset provided in the StatLib library. In line with its use by Ross Quinlan (1993) in predicting the mpg attribute, eight of the original instances were removed because...

Importing the data

We are going to start by importing one of the required libraries for this task: NumPy. Let's get started!

1. We are also going to import pathlib, matplotlib, SeaBorn, tensorFlow, and keras. We've already learned about TensorFlow and Keras. matplotlib and SeaBorn are used for visualization. pathlib provides a readable and easier way to build paths. Finally, pandas is one of the best data preprocessing libraries available:

In[1]: import pathlib
In[2]: import matplotlib.pyplot as plt
In[3]: import pandas as pd
In[4]: import seaborn as sns
In[5]: import tensorflow as tf
In[6]: from tensorflow import keras
In[7]: from tensorflow.keras import layers
In[8]: from __future__ import absolute_import, division, print_function, unicode_literals

2. Now, we will import the data using https://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data:

In[9]: dataset_path = keras.utils.get_file("auto-mpg.data", "https://archive.ics.uci.edu...

Splitting the data

It's time to split the data into train/test sets. Bear in mind that sometimes, people like to split their data three ways; train, test, and validation. For now, though, we'll keep things simple and just use train and test. 

First, we will split the data into train_data and test_data. We are going to use train_data for training and test_data for prediction. We are going to have an 80-20 split:

In[19]: train_data = dataset.sample(frac=0.8, random_state=0)

In[20]: test_data = dataset.drop(train_dataset.index)

Now, we will separate the MPG label from the train and test data:

In[21]: train_labels = train_data.pop('MPG')

In[22]: test_labels = test_data.pop('MPG')

In the next section, we will normalize the dataset as this helps us improve the performance of the model.

Standardizing the data

Usually, when we use neural networks, we get improved performance when we standardize the data. Standardization just means normalizing the values so that they all fit between a certain range, such as 0 to 1 or -1 to +1.

The scikit-learn library also provides a nice function for this. Click on the following link for more information: http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html.

There is one more way to normalize the data: by using mean and standard deviations. The normalization function in the following code can standardize the data for better performance:

In[23]: def normalization(x):
          return (x - train_stats['mean']) / train_stats['std']
In[24]: normed_train_data = normalization(train_dataset)
In[25]: normed_test_data = normalization(test_dataset)

Previously, we performed standardization using mean and standard deviation. In general, it is recommended to normalize the data in one go...

Building and compiling the model

Now, let's build a simple neural network.

In this section, we will add the layers we'll use in our deep learning model. Let's get started:

1. First, we will import tensorflow, keras, and layers: 

In[26]: import tensorflow as tf
In[27]: from tensorflow import keras
In[28]: from tensorflow.keras import layers

2. Now, we can build our model. First, we are going to use Sequential() with two hidden layers and output a single continuous value. We have a wrapper function called model_building for this. When we compile the model, we need to choose a loss function, an optimizer, and accuracy metrics. We used RMSprop as the optimizer, mean_square_error as the loss function, and mean_absolute_error and mean_square_error as the required metrics. Mean Squared Error (MSE) is a common loss function used for regression problems. Evaluation metrics for regression is a Mean Absolute Error (MAE...

Training the model

In this section, we will train the model.

Here, we can see that model.fit helps us start the training process:

# Display training progress by printing a single dot for each completed epoch
In[32]: class PrintDot(keras.callbacks.Callback):
         def on_epoch_end(self, epoch, logs):
            if epoch % 100 == 0: print('')
            print('.', end='')

In[33]: EPOCHS = 1000

In[34]: history = model.fit(normed_train_data, train_labels,
  epochs=EPOCHS, validation_split = 0.2, verbose=0,
  callbacks=[PrintDot()])

Now, we will visualize the model's training progress:

In[35]: hist = pd.DataFrame(history.history)
In[36]: hist['epoch'] = history.epoch
In[37]: hist.tail()
In[38]: def plot_training_history(history):
          hist = pd.DataFrame(history.history)
          hist['epoch'] = history.epoch

          plt.figure()
          plt.xlabel('Epoch')
          plt.ylabel('Mean Abs Error [MPG]')
   ...

Predicting new, unseen data

Now, let's see how we did by making predictions on the test data of the Duto-Mpg dataset. Remember, our model has never seen the test data that we scaled previously! This process is the same process that you would use on brand-new data.

Let's look at the test data that we've just analyzed:

test_predictions = model.predict(normed_test_data).flatten()

plt.scatter(test_labels, test_predictions)
plt.xlabel('True Values [MPG]')
plt.ylabel('Predictions [MPG]')
plt.axis('equal')
plt.axis('square')
plt.xlim([0,plt.xlim()[1]])
plt.ylim([0,plt.ylim()[1]])
_ = plt.plot([-100, 100], [-100, 100])
plt.show()

The scatter plot between the predicted and true values shows the error in the model:

Fig 3.11: True values versus predicted values

In the next section, we will evaluate the performance of the model.

Evaluating the model's performance

So, how well did we do? How do we actually measure how well we did? It all depends on the situation. 

Let's evaluate our model by plotting the error counts:

error = test_predictions - test_labels
plt.hist(error, bins = 25)
plt.xlabel("Prediction Error [MPG]")
_ = plt.ylabel("Count")
plt.show()

Now, let's view the output:

Fig 3.12: Count of predicted errors in the model

It looks like the model predicted reasonably well. The distribution error of the model shows it is not quite Gaussian or normally distributed, but we can expect non-Gaussian as the number of samples is very small. 

Saving and loading models

Now that we have trained the model, we need to save and load it. Let's get started:

1. We will start by saving the model:

In[31]: model.save('myfirstmodel.h5')\

2. Next, we will import the model:

In[32]: from keras.models import load_model
In[33]: newmodel = tf.keras.models.load_model('myfirstmodel.h5')

3. Finally, we will predict the imported model:

In[34]: test_predictions = model.predict(normed_test_data).flatten()

With that, you have implemented your first deep learning model!

Summary

In this chapter, we began by understanding the basics of Keras and saw why Keras is so useful. We learned about the types of Keras execution, and we also built our first deep learning model step by step. We went through the different steps of building a model: importing data, splitting data, normalizing data, building the model, compiling the model, training the model, predicting unseen data, evaluating model performance, and finally saving and loading the model. 

In the next chapter, we are going to learn about computer vision techniques.