Introducing Neural Nets for Predictive Analytics

In the last two chapters, we have presented some of the most basic and popular models for regression and classification tasks. In this chapter, we introduce a family of models based on neural networks. This family of models is the basis for the field of deep learning—an approach to machine learning behind some of the most exciting and recent advances in the field of artificial intelligence.

This chapter will give you enough knowledge to be able to use neural networks for predictive analytics; the point here is to present the fundamental concepts about these models and learn to train the most fundamental type of neural network—the multilayer perceptron (MLP).

First, we will cover the main concepts of neural networks when talking about the anatomy of an MLP; then we will discuss how these models learn to make predictions...

Technical requirements

• Python 3.6 or higher
• Jupyter Notebooks
• Recent versions of the following Python libraries: NumPy, pandas, matplotlib, Seaborn, and scikit-learn
• Recent installations of TensorFlow and Keras

Introducing neural network models

There is no question that lately neural networks and deep learning are terms that have attracted a lot of attention. Although there is definitely a lot of hype and misunderstanding of these technologies, they are behind some of the most important developments and breakthroughs in the field of artificial intelligence—self-driving cars, language translators, speech recognition, super-human level players in many board games, computer vision, and many other achievements are directly related to different kinds of deep learning models.

In this chapter, we will learn about one basic type of neural network model—the MLPs will use these models to solve predictive analytics problems, in particular, we will apply them to solve the two examples we have been working on within the book. After finishing this chapter, we will be able to include...

Introducing TensorFlow and Keras

Now we know that neural networks are a special type of machine learning model. Although, usually these models need huge amounts of data to start outperforming other machine learning approaches, one big advantage is that the process of training neural networks can make use of parallelization in hardware such as graphical processing units (GPUs), which do the operations needed for training neural networks faster than traditional CPUs. This is the reason that in the past few years, new specialized software frameworks have been developed with the capacity to make use of GPUs; examples of these frameworks are Theano, Caffe, and TensorFlow. These frameworks have allowed the deep learning models to be used for professionals outside specialized academic circles, thus democratizing the use of these powerful models. In this section, we introduce the two...

Regressing with neural networks

We will again use our diamonds dataset. Although this is a small dataset and MLP is perhaps a model that is too complicated for this problem, there is no reason we could not use an MLP to solve it; in addition to this, remember that back when we defined the hypothetical problem, we established that the stakeholders wanted a model that was as accurate as possible in their predictions, so let's see how accurate we can get with an MLP. As always, let's import the libraries we will use:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import os
%matplotlib inline

Now, since we are beginning from scratch, load and prepare the dataset:

DATA_DIR = '../data'
FILE_NAME = 'diamonds.csv'
data_path = os.path.join(DATA_DIR, FILE_NAME)
diamonds = pd.read_csv(data_path)
## Preparation done from Chapter...

Classification with neural networks

Now, let's perform our classification task using a neural network. As you will see, the only change necessary in an MLP for it to be able to perform classification is in the output layer:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import os
%matplotlib inline

As always, let's start from scratch and import and prepare our data:

# Loading the dataset
DATA_DIR = '../data'
FILE_NAME = 'credit_card_default.csv'
data_path = os.path.join(DATA_DIR, FILE_NAME)
ccd = pd.read_csv(data_path, index_col="ID")
ccd.rename(columns=lambda x: x.lower(), inplace=True)
ccd.rename(columns={'default payment next month':'default'}, inplace=True)

# getting the groups of features
bill_amt_features = ['bill_amt'+ str(i) for i in range(1,7)]
pay_amt_features = [&apos...

The dark art of training neural networks

From the results we got, we can see that there is a clear symptom of overfitting—the training accuracy looks great (91%), but the testing accuracy is lower than even random guessing. The most likely two causes for this are as follows:

• The model has too many parameters
• The model has been trained for too long

Since we are overfitting, we need to try some regularization technique; the simplest in the case of neural networks is training the model for fewer epochs. Now, let's get the initial weights and biases of the network back to the original values (the ones we saved in the a.h5 file):

nn_classifier.load_weights('class_initial_w.h5')

Now the weights have been reset, let's train our model again, this time only for 50 epochs:

batch_size = 64
n_epochs = 50
nn_classifier.compile(loss='binary_crossentropy&apos...

Summary

In this chapter, we introduced the most fundamental type of deep learning model—the MLP. We covered a lot of new concepts related to this power class of models such as deep learning, neural network models, and the activation functions of neurons. We also learned about TensorFlow, which is a framework to train deep learning models; we used it as a backend for running the calculations necessary to train our models. We covered Keras, where we first build a network, and then we compile it (indicating the loss and optimizer), and finally, we train the model. Lastly, we covered dropout, which is a regularization technique that is often used with neural networks, although it works best with very large networks. To conclude, neural networks are hard to train because they involve making many decisions; a lot of practice and knowledge is needed to be able to use these models...

Further reading

• Bengio, Y (2012). Practical recommendations for gradient-based training of deep architectures. Neural networks: Tricks of the trade (pp. 437-478).
• Chollet, F (2017). Deep Learning with Python, Manning Publications.
• Glorot, X, and Bengio, Y (2010, March). Understanding the difficulty of training deep feedforward neural networks. From the proceedings of the 13^th international conference on artificial intelligence and statistics (pp. 249-256).
• Keskar, N. S., et al. (2016). On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836.