In the previous chapter, it was explained why deep learning has become so popular nowadays, and PyTorch was introduced as one of the most popular libraries for developing deep learning solutions. Although the main syntax for building a neural network using PyTorch was explained, in this chapter, we will further explore the concept of neural networks.

Although neural network theory was developed several decades ago, since the concept evolved from the notion of the perceptron, different architectures have been created to solve different data problems in recent times. This is, in part, due to the different data formats that can be found in real-life data problems, such as text, audio, and images.

The purpose of this chapter is to dive into the topic of neural networks and their main advantages and disadvantages so that you can understand when and how to use them. Then, we will explain the building blocks of the most popular neural network architectures: artificial neural networks (ANNs), convolutional neural networks (CNNs), and recurrent neural networks (RNNs).

Following this, the process of building an effective model will be explained by solving a real-life regression problem. This includes preparing the data to be fed to the neural network (also known as data preprocessing), defining the neural network architecture to be used, and evaluating the performance of the model, with the objective of determining how it can be improved to achieve an optimal solution.

The aforementioned process will be done using one of the neural network architectures that will be discussed in this chapter, all while taking into consideration that the solution for each data problem should be carried out using the architecture that performs best for the data type in question. The other architectures will be used in subsequent chapters to solve more complicated data problems that involve using images and sequences of text as input data.

Note

All the code present in this chapter can be found at: https://packt.live/34MBauE.

Neural networks learn from training data, rather than being programmed to solve a particular task by following a set of rules. This learning process can follow one of the following methodologies:

• Supervised learning: This is the simplest form of learning as it consists of a labeled dataset, where the neural network finds patterns that explain the relationship between the features and the target. The iterations during the learning process aim to minimize the difference between the predicted value and the ground truth. One example of this is classifying a plant based on the attributes of its leaves.
• Unsupervised learning: In contrast to the preceding methodology, unsupervised learning consists of training a model with unlabeled data (meaning that there is no target value). The purpose of this is to arrive at a better understanding of the input data. In general, networks take input data, encode it, and then reconstruct the content from the encoded version, ideally keeping the relevant information. For instance, given a paragraph, a neural network can map the words and then suggest which ones are the most important or descriptive for the paragraph. These can then be used as tags.
• Reinforcement learning: This methodology consists of learning from the input data, with the main objective of maximizing a reward function in the long run. This is achieved by learning from the data as it comes in, rather than it being trained over static data (as in supervised learning). Hence, decisions are not made based on the immediate reward, but on the accumulation of it in the entire learning process. An example of this is a model that allocates resources to different tasks, with the objective of minimizing bottlenecks that slow down general performance.

  Note

  From the learning methodologies we've mentioned here, the most commonly used one is supervised learning, which is the one that will be mainly used in subsequent sections. This means that all the exercises, activities, and examples in this chapter will use a labeled dataset as input data.

What Are Neural Networks?

As we discussed earlier, neural networks are a type of machine learning algorithm that's modeled on the anatomy of the human brain and that uses mathematical equations to learn a pattern from the observations that were made from the training data.

However, to actually understand the logic behind the training process that neural networks typically follow, it is important to understand the concept of perceptrons.

Developed during the 1950s by Frank Rosenblatt, a perceptron is an artificial neuron that takes several inputs and produces a binary output, similar to neurons in the human brain. This then becomes the input of a subsequent perceptron (neuron). Perceptrons are the essential building blocks of a neural network (just like neurons are the building blocks of the human brain):

Figure 2.1: Diagram of a perceptron

Here, X1, X2, X3, and X4 represent the different inputs of the perceptron, and there could be any number of these. The circle is the perceptron, which is where the inputs are processed to arrive at an output.

Rosenblatt also introduced the concept of weights (w1, w2, …, wn), which are numbers that express the importance of each input. The output can be either 0 or 1, and it depends on whether the weighted sum of the inputs is above or below a given threshold (a numerical limit set by the developer or by a constraint of the data problem), which can be set as a parameter of the perceptron, as can be seen here:

Figure 2.2: Equation for the output of perceptrons

Exercise 2.01: Performing the Calculations of a Perceptron

The following exercise does not require programming of any kind; instead, it consists of simple calculations to help you understand the notion of the perceptron. To perform these calculations, consider the following scenario.

There is a music festival in your town next Friday, but you are ill and trying to decide whether to go (where 0 means you are not going and 1 means you are going). Your decision relies on three factors:

• Will there be good weather? (X1)
• Do you have someone to go with? (X2)
• Is the music to your liking? (X3)

For the preceding factors, we will use 1 if the answer to the question is yes, and 0 if the answer is no. Additionally, since you are very sick, the factor related to the weather is highly relevant, and you decide to give this factor a weight twice as big as the other two factors. Hence, you decide that the weights for the factors will be 4 (w1), 2 (w2), and 2 (w3). Now, consider a threshold of 5:

1. With the information provided, calculate the output of the perceptron when considering that the weather is not good next Friday, but that you have someone to go with and you like the music at the festival:

Figure 2.3: Output of the perceptron

Considering that the output is less than the threshold, the final result will be equal to 0, meaning that you should not go to the festival to avoid the risk of getting even more ill.

You have successfully performed the calculations of a perceptron, which is the starting point of understanding the learning process that occurs inside neural networks.

Multi-Layer Perceptron

Considering what we learned about in the previous section, the notion of a multi-layered network consists of a network of multiple perceptrons stacked together (also known as nodes or neurons), such as the one shown here:

Figure 2.4: Diagram of a multi-layer perceptron

Note

The conventional way to refer to the layers in a neural network is as follows:

The first layer is the input layer, the last layer is the output layer, and all the layers in between are hidden layers.

Here, again, a set of inputs is used to train the model, but instead of feeding a single perceptron, they are fed to all the perceptrons (neurons) in the first layer. Next, the outputs that are obtained from this layer are used as inputs for the perceptrons in the subsequent layer and so on until the final layer is reached, which is in charge of outputting a result.

Note that the first layer of a perceptron handles a simple decision process by weighting the inputs, while the subsequent layer can handle more complex and abstract decisions based on the output of the previous layer, and hence the state-of-the-art performance of deep neural networks (networks that use many layers) for complex data problems.

Different to conventional perceptrons, neural networks have evolved to have one or multiple nodes in the output layer so that they are able to present the result either as binary or multiclass.

The Learning Process of a Neural Network

In general terms, a neural network is made up of multiple neurons, where each neuron computes a linear function, along with an activation function, to arrive at an output based on some inputs (an activation function is designed to break linearity – this will be explained in more detail later in this chapter). This output is tied to a weight, which represents its level of importance, and will be used for calculations in the following layer.

Moreover, these calculations are carried out throughout the entire architecture of the network, until a final output is reached. This output is used to determine the performance of the network in comparison to the ground truth, which is then used to adjust the different parameters of the network to start the calculation process over again.

Considering this, the training process of a neural network can be seen as an iterative process that goes forward and backward through the layers of the network to arrive at an optimal result, which can be seen in the following diagram (loss functions will be covered later in this chapter):

Figure 2.5: Diagram of the learning process of a neural network

Forward Propagation

This is the process of going from left to right through the architecture of the network while performing calculations using the input data to arrive at a prediction that can be compared to the ground truth. This means that every neuron in the network will transform the input data (the initial data or data received from the previous layer) according to the weights and biases that it has associated with it and will send the output to the subsequent layer until a final layer is reached and a prediction is made.

Note

In neural networks, biases are numerical values that help shift the activation function of each neuron in order to avoid zero values that may affect the training process. Their role in the training of neural networks will be explained later in this chapter.

The calculations that are performed in each neuron include a linear function that multiplies the input data by some weight plus a bias, which is then passed through an activation function. The main purpose of the activation function is to break the linearity of the model, which is crucial considering that most real-life data problems that are solved using neural networks are not defined by a line, but rather by a complex function. These formulas are as follows:

Figure 2.6: Calculations performed by each neuron

Here, as we mentioned previously, X refers to the input data, W is the weight that determines the level of importance of the input data, b is the bias value, and sigma () represents the activation function that's applied over the linear function.

The activation function serves the purpose of introducing non-linearity to the model. There are different activation functions to choose from, and a list of the ones most commonly used nowadays is as follows:

• Sigmoid: This is S-shaped, and it basically converts values into simple probabilities between 0 and 1, where most of the outputs that are obtained by the sigmoid function will be close to the extremes of 0 and 1:

Figure 2.7: Sigmoid activation function

The following plot shows the graphical representation of the sigmoid activation function:

Figure 2.8: Graphical representation of the sigmoid activation function

• Softmax: Similar to the sigmoid function, this calculates the probability distribution of an event over n events, meaning that its output is not binary. In simple terms, this function calculates the probability of the output being one of the target classes in comparison to the other classes:

Figure 2.9: Softmax activation function

Considering that its output is a probability, this activation function is often found in the output layer of classification networks.

• Tanh: This function represents the relationship between the hyperbolic sine and the hyperbolic cosine, and the result is between -1 and 1. The main advantage of this activation function is that negative values can be dealt with more easily:

Figure 2.10: Tanh activation function

The following plot shows the graphical representation of the tanh activation function:

Figure 2.11: Graphical representation of the tanh activation function

• Rectified Linear Function (ReLU): This basically activates a node given that the output of the linear function is above 0; otherwise, its output will be 0. If the output of the linear function is above 0, the result from this activation function will be the raw number it received as input:

Figure 2.12: ReLU activation function

Conventionally, this activation function is used for all hidden layers. We will learn more about hidden layers in the upcoming sections of this chapter. The following plot shows the graphical representation of the ReLU activation function:

Figure 2.13: Graphical representation of the ReLU activation function

The Calculation of Loss Functions

Once forward propagation is complete, the next step in the training process is to calculate a loss function to estimate the error of the model by comparing how good or bad the prediction is in relation to the ground truth value. Considering this, the ideal value to be reached is 0, which would mean that there is no divergence between the two values.

This means that the goal in each iteration of the training process is to minimize the loss function by changing the parameters (weights and biases) that are used to perform the calculations during the forward pass.

Again, there are multiple loss functions to choose from. However, the most commonly used loss functions for regression and classification tasks are as follows:

• Mean squared error (MSE): Widely used to measure the performance of regression models, the MSE function calculates the sum of the distance between the ground truth and the prediction values:

Figure 2.14: MSE loss function

Here, n refers to the number of samples, is the ground truth values, and is the predicted value.

• Cross-entropy/multi-class cross-entropy: This function is conventionally used for binary or multi-class classification models. It measures the divergence between two probability distributions; a large loss function will represent a large divergence. Hence, the objective here is to also minimize the loss function:

Figure 2.15: Cross-entropy loss function

Again, n refers to the number of samples. and are the ground truth and the predicted value, respectively.

Backward Propagation

The final step in the training process consists of going from right to left in the architecture of the network to calculate the partial derivatives (also known as gradients) of the loss function in respect to the weights and biases in each layer in order to update these parameters (weights and biases) so that in the next iteration step, the loss function is lower.

The final objective of the optimization algorithm is to find the global minima where the loss function has reached the least possible value, as shown in the following plot:

Note

A local minima refers to the smallest value within a section of the function domain. On the other hand, a global minima refers to the smallest value of the entire domain of the function.

Figure 2.16: Loss function optimization through the iteration steps in a two-dimensional space

Here, the dot furthest to the left, A, is the initial value of the loss function before any optimization. The dot furthest to the right, B, at the bottom of the curve, is the loss function after several iteration steps, where its value has been minimized. The process of going from one dot to another is called a step.

However, it is important to mention that the loss function is not always as smooth as the preceding one, which can introduce the risk of reaching a local minima during the optimization process.

This process is also called optimization, and there are different algorithms that vary in methodology to achieve the same objective. The most commonly used optimization algorithm will be explained next.

Gradient Descent

Gradient descent is the most widely used optimization algorithm among data scientists, and it is the basis of many other optimization algorithms. After the gradients for each neuron are calculated, the weights and biases are updated in the opposite direction of the gradient, which should be multiplied by a learning rate (used to control the size of the steps taken in each optimization), as shown in the following equations.

The learning rate is crucial during the training process as it prevents the update of the weights and biases from over/undershooting, which may prevent the model from reaching convergence or delay the training process, respectively.

The optimization of weights and biases in the gradient descent algorithm is as follows:

Figure 2.17: Optimization of parameters in the gradient descent algorithm

Here, α refers to the learning rate, and dw/db represents the gradients of the weights or biases in a given neuron. The product of the two values is subtracted from the original value of the weight or bias in order to penalize the higher values, which are contributing to computing a large loss function.

An improved version of the gradient descent algorithm is called stochastic gradient descent, and it basically follows the same process, with the distinction that it takes the input data in random batches instead of in one chunk, which improves the training times while reaching outstanding performance. Moreover, this approach allows for the use of larger datasets because by using small batches of the dataset as inputs, we are no longer limited by computational resources.

Advantages and Disadvantages

The following is an explanation of the advantages and disadvantages of neural networks.

Advantages

Neural networks have become increasingly popular in the last few years for four main reasons:

• Data: Neural networks are widely known for their ability to capitalize on large amounts of data, and thanks to the advances in hardware and software, the collection and storage of massive databases is now possible. This has allowed neural networks to show their real potential as more data is fed into them.
• Complex data problems: As we explained previously, neural networks are excellent for solving complex data problems that cannot be tackled by other machine learning algorithms. This is mainly due to their ability to process large datasets and uncover complex patterns.
• Computational power: Advances in technology have also increased the computational power that's available these days, which is crucial for training neural network models that use millions of pieces of data.
• Academic research: Thanks to the preceding three points, a proliferation of academic research on this topic is available on the internet, which not only facilitates the immersion of new research each day, but also helps keep the algorithms and hardware/software requirements up to date.

Disadvantages

Just because there are a lot of advantages to using a neural network does not mean that every data problem should be solved this way. This is a mistake that is commonly made. There is no one algorithm that will perform well for all data problems, and selecting the algorithm to use should depend on the resources available, as well as the data problem.

Although neural networks are thought to outperform almost any machine learning algorithm, it is crucial to consider their disadvantages as well so that you can weigh up what matters most for the data problem. Let's go through them now:

• Black box: This is one of the most commonly known disadvantages of neural networks. It basically means that how and why a neural network reached a certain output is unknown. For instance, when a neural network incorrectly predicts a cat picture as a dog, it is not possible to know what the cause of the error was.
• Data requirements: The vast amounts of data that they require to achieve optimal results can be equally an advantage and a disadvantage. Neural networks require more data than traditional machine learning algorithms, which can be the main reason to choose between them and other algorithms for some data problems. This becomes a greater issue when the task at hand is supervised, which means that the data needs to be labeled.
• Training times: Tied to the preceding disadvantage, the need for vast amounts of data also makes the training process last longer than traditional machine learning algorithms, which, in some cases, is not an option. Training times can be reduced through the use of GPUs, which speed up computation.
• Computationally expensive: Again, the training process of neural networks is computationally expensive. While one neural network could take weeks to converge, other machine learning algorithms could take hours or minutes to be trained. The amount of computational resources needed depends on the quantity of data at hand, as well as the complexity of the network; deeper neural networks take a longer time to train.

  Note

  There are a wide variety of neural network architectures. Three of the most commonly used ones will be explained in this chapter, along with their practical implementation in subsequent chapters. However, if you wish to learn about other architectures, visit http://www.asimovinstitute.org/neural-network-zoo/.

Introduction to Artificial Neural Networks

Artificial neural networks (ANNs), also known as multi-layer perceptrons, are collections of multiple perceptrons. The connection between perceptrons occurs through layers. One layer can have as many perceptrons as desired, and they are all connected to all the other perceptrons in the preceding and subsequent layers.

Networks can have one or more layers. Networks with over four layers are considered to be deep neural networks and are commonly used to solve complex and abstract data problems.

ANNs are typically composed of three main elements, which were explained earlier, and can also be seen in the following image:

1. Input layer: This is the first layer of the network, conventionally located furthest to the left in the graphical representation of a network. It receives the input data before any calculation is performed and completes the first set of calculations. This is where the most generic patterns are uncovered.

   For supervised learning problems, the input data consists of a pair of features and targets. The job of the network is to uncover the correlation or dependency between the features and target.

2. Hidden layers: Next, the hidden layers can be found. A neural network can have many hidden layers, meaning there can be any number of layers between the input layer and the output layer. The more layers it has, the more complex data problems it can tackle, but it will also take longer to train. There are also neural network architectures that do not contain hidden layers at all, which is the case with single-layer networks.

   In each layer, a computation is performed based on the information that's received as input from the previous layer, which is then used to output a value that will become the input of the subsequent layer.

3. Output layer: This is the last layer of the network as is located at the far right of the graphical representation of the network. It receives data after the data has been processed by all the neurons in the network to make a final prediction.

   The output layer can have one or more neurons. The former refers to models where the solution is binary, in the form of 0s or 1s. On the other hand, the latter case consists of models that output the probability of an instance belonging to each of the possible class labels (the possible values that the target variable has), meaning that the layer will have as many neurons as there are class labels:

Figure 2.18: Architecture of a neural network with two hidden layers

Introduction to Convolutional Neural Networks

Convolutional Neural Networks (CNNs) are mostly used in the field of computer vision, where, in recent decades, machines have achieved levels of accuracy that surpass human ability.

CNNs create models that use subgroups of neurons to recognize different aspects of an image. These groups should be able to communicate with each other so that, together, they can form the complete image.

Considering this, the layers in the architecture of a CNN divide their recognition tasks. The first layers focus on trivial patterns, while the layers at the end of the network use that information to uncover more complex patterns.

For instance, when recognizing human faces in pictures, the first couple of layers focus on finding edges that separate one feature from another. Next, the subsequent layers emphasize certain features of the face, such as the nose. Finally, the last couple of layers use this information to put the entire face of the person together.

This idea of activating a group of neurons when certain features are encountered is achieved through the use of filters (kernels), which are one of the main building blocks of the architecture of CNNs. However, they are not the only elements present in the architecture, which is why a brief explanation of all the components of CNNs will be provided here:

Note

The concepts of padding and stride, which you might have heard of when using CNNs, will be explained in subsequent chapters of this book.

1. Convolutional layers: In these layers, a convolutional computation occurs between an image (represented as a matrix of pixels) and a filter. This computation produces a feature map as output that ultimately serves as input for the next layer.

   The computation takes a subsection of the image matrix of the same shape of the filter and performs a multiplication of the values. Then, the sum of the product is set as the output for that section of the image, as shown in the following diagram:

   

   Figure 2.19: Convolution operation between the image and filter

   Here, the matrix to the left is the input data, the matrix in the middle is the filter, and the matrix to the right is the output from the computation. The computation that occurred with the values highlighted by the boxes can be seen here:

   

   Figure 2.20: Convolution of the first section of the image

   This convolutional multiplication is done for all the subsections of the image. The following diagram shows another convolution step for the same example:

   

   Figure 2.21: A further step in the convolution operation

   One important notion of convolutional layers is that they are invariant in such a way that each filter will have a specific function, which does not vary during the training process. For instance, a filter in charge of detecting ears will only specialize in that function throughout the training process.

   Moreover, a CNN will typically have several convolutional layers, considering that each of them will focus on identifying a particular feature or set of features of the image, depending on the filters that are used. Commonly, there is one pooling layer between two convolutional layers.

2. Pooling layers: Although convolutional layers are capable of extracting relevant features from images, their results can become enormous when analyzing complex geometrical shapes, which would make the training process impossible in terms of computational power, hence the invention of pooling layers.

   These layers not only accomplish the goal of reducing the output of the convolutional layers, but also achieve the removal of any noise that's present in the features that have been extracted, which ultimately helps to increase the accuracy of the model.

   There are two main types of pooling layers that can be applied, and the idea behind them is to detect the areas that express a stronger influence in the image so that the other areas can be overlooked.

   Max pooling: This operation consists of taking a subsection of the matrix of a given size and taking the maximum number in that subsection as the output of the max pooling operation:

   

   Figure 2.22: A max pooling operation

   In the preceding diagram, by using a 3 x 3 max pooling filter, the result on the right is achieved. Here, the yellow section (top-left corner) has a maximum number of 4, while the orange section (top-right corner) has a maximum number of 5.

   Average pooling: Similarly, the average pooling operation takes subsections of the matrix and takes the number that meets the rule as output, which, in this case, is the average of all the numbers in the subsection in question:

   

   Figure 2.23: An average pooling operation

   Here, using a 3 x 3 filter, we get 2.9, which is the average of all the numbers in the yellow section (top-left corner), while 3.2 is the average for the ones in the orange section (top-right corner).

3. Fully connected layers: Finally, considering that the network would be of no use if it was only capable of detecting a set of features without having the capability of classifying them into a class label, fully connected layers are used at the end of CNNs to take the features that were detected by the previous layer (known as the feature map) and output the probability of that group of features belonging to a class label, which is used to make the final prediction.

   Like ANNs, fully connected layers use perceptrons to calculate an output based on a given input. Moreover, it is crucial to mention that CNNs typically have more than one fully connected layer at the end of the architecture.

By combining all of these concepts, the conventional architecture of CNNs is obtained. There can be as many layers of each type as desired, and each convolutional layer can have as many filters as desired (each for a particular task). Additionally, the pooling layer should have the same number of filters as the previous convolutional layer, as shown in the following image:

Figure 2.24: Diagram of the CNN architecture

Introduction to Recurrent Neural Networks

The main limitation of the aforementioned neural networks (ANNs and CNNs) is that they learn only by considering the current event (the input that is being processed) without taking into account previous or subsequent events, which is inconvenient considering that we humans do not think that way. For instance, when reading a book, you can understand each sentence better by considering the context from the previous paragraph or more.

Due to this, and taking into account the fact that neural networks aim to optimize several processes that are traditionally done by humans, it is crucial to think of a network that's able to consider a sequence of inputs and outputs, hence the creation of recurrent neural networks (RNNs). They are a robust type of neural network that allow solutions to be found for complex data problems through the use of internal memory.

Simply put, these networks contain loops in them that allow for the information to remain in their memory for longer periods, even when a subsequent set of information is being processed. This means that a perceptron in an RNN not only passes over the output to the following perceptron, but it also retains a bit of information to itself, which can be useful for analyzing the next bit of information. This memory-keeping capability allows them to be very accurate in predicting what's coming next.

The learning process of an RNN, similar to other networks, tries to map the relationship between an input (x) and an output (y), with the difference being that these models also take into consideration the entire or partial history of previous inputs.

RNNs allow sequences of data to be processed in the form of a sequence of inputs, a sequence of outputs, or even both at the same time, as shown in the following diagram:

Figure 2.25: Sequence of data handled by RNNs

Here, each box is a matrix and the arrows represent a function that occurs. The bottom boxes are the inputs, the top boxes are the outputs, and the middle boxes represent the state of the RNN at that point, which holds the memory of the network.

From left to right, the preceding diagrams can be explained as follows:

1. A typical model that does not require an RNN to be solved. It has a fixed input and a fixed output. This can refer to image classification, for instance.
2. This model takes in an input and yields a sequence of outputs. Take, for instance, a model that receives an image as input; the output should be an image caption.
3. Contrary to the preceding model, this model takes a sequence of inputs and yields a single outcome. This type of architecture can be seen on sentiment analysis problems, where the input is the sentence to be analyzed and the output is the predicted sentiment behind the sentence.
4. The final two models take a sequence of inputs and return a sequence of outputs, with the difference being that the first one analyzes the inputs and generates the outputs at the same time; for example, when each frame of a video is being labeled individually. On the other hand, the second many-to-many model analyzes the entire set of inputs in order to generate the set of outputs. An example of this is language translation, where the entire sentence in one language needs to be understood before proceeding with the actual translation.

The first step in the development of any deep learning model – after gathering the data, of course – should be preparation of the data. This is crucial if we wish to understand the data at hand to outline the scope of the project correctly.

Many data scientists fail to do so, which results in models that perform poorly, and even models that are useless as they do not answer the data problem to begin with.

The process of preparing the data can be divided into three main tasks:

1. Understanding the data and dealing with any potential issues
2. Rescaling the features to make sure no bias is introduced by mistake
3. Splitting the data to be able to measure performance accurately

All three tasks will be further explained in the next section.

Note

All of the tasks we explained previously are pretty much the same when applying any machine learning algorithm, considering that they refer to the techniques that are required to prepare data beforehand.

Dealing with Messy Data

This task mainly consists of performing exploratory data analysis (EDA) to understand the data available, as well as to detect potential issues that may affect the development of the model.

The EDA process is useful as it helps the developer uncover information that's crucial to the definition of the course of action. This information is explained here:

1. Quantity of data: This refers both to the number of instances and the number of features. The former is crucial for determining whether it is necessary or even possible to solve the data problem using a neural network, or even a deep neural network, considering that such models require vast amounts of data to achieve high levels of accuracy. The latter, on the other hand, is useful for determining whether it would be a good practice to develop some feature selection methodologies beforehand in order to reduce the number of features, to simplify the model, and to eliminate any redundant information.
2. The target feature: For supervised models, data needs to be labeled. Considering this, it is highly important to select the target feature (the objective that we want to achieve by building the model) in order to assess whether the feature has many missing or outlier values. Additionally, this helps determine the objective of the development, which should be in line with the data that's available.
3. Noisy data/outliers: Noisy data refers to values that are visibly incorrect, for instance, a person who is 200 years old. On the other hand, outliers refer to values that, although they may be correct, are very far from the mean, for instance, a 10-year-old college student.

   There is not an exact science for detecting outliers, but there are some methodologies that are commonly accepted. Assuming a normally distributed dataset, one of the most popular ones is determining any value that is about 3-6 standard deviations away from the mean as an outlier.

   An equally valid approach to identifying outliers is to select those values at the 99th and 1st percentiles.

   It is very important to handle such values when they represent over 5% of the data for a feature because failing to do so may introduce bias to the model. The way to handle these values, as with any other machine learning algorithm, is to either delete the outlier values or assign new values using mean or regression imputation techniques.

4. Missing values: Similar to the aforementioned, a dataset with many missing values can introduce bias to the model, considering that different models will make different assumptions about those values. Again, when missing values represent over 5% of the values of a feature, they should be handled by eliminating or replacing them, again using the mean or regression imputation techniques.
5. Qualitative features: Finally, checking whether the dataset contains qualitative data is also a key step, considering that removing or encoding data may result in more accurate models.

   Additionally, in many research developments, several algorithms are tested on the same data in order to determine which one performs better, and some of these algorithms do not tolerate the use of qualitative data, as is the case with neural networks. This proves the importance of converting or encoding them to be able to feed all the algorithms the same data.

Exercise 2.02: Dealing with Messy Data

Note

All of the exercises in this chapter will be completed using the Appliances energy prediction Dataset sourced from the UC Irvine Machine Learning Repository, which was downloaded from https://archive.ics.uci.edu/ml/datasets/Appliances+energy+prediction. It can also be found in this book's GitHub repository: https://packt.live/34MBoSw

The Appliances energy prediction Dataset contains 4.5 months of data related to temperature and humidity measures for different rooms in a low-energy building, with the objective of predicting the energy that's used by certain appliances.

In this exercise, we will use pandas, which is a popular Python package, to explore the data at hand and learn how to detect missing values, outliers, and qualitative values. Perform the following steps to complete this exercise:

Note

For the exercises and activities within this chapter, you will need to have Python 3.7, Jupyter 6.0, NumPy 1.17, and Pandas 0.25 installed on your local machine.

1. Open a Jupyter notebook to implement this exercise.
2. Import the pandas library:

   import pandas as pd

3. Use pandas to read the CSV file containing the dataset we downloaded from the UC Irvine Machine Learning Repository site.

   Next, drop the column named date as we do not want to consider it for the following exercises:

   data = pd.read_csv("energydata_complete.csv")
   data = data.drop(columns=["date"])

   Finally, print the head of the DataFrame:

   data.head()

   The output should look as follows:

   

   Figure 2.26: Top instances of the Appliances energy prediction dataset

4. Check for categorical features in your dataset:

   cols = data.columns
   num_cols = data._get_numeric_data().columns
   list(set(cols) - set(num_cols))

   The first line generates a list of all the columns in your dataset. Next, the columns that contain numeric values are stored in a variable as well. Finally, by subtracting the numeric columns from the entire list of columns, it is possible to obtain those that are not numeric.

   The resulting list is empty, which indicates that there are no categorical features to deal with.

5. Use Python's isnull() and sum() functions to find out whether there are any missing values in each column of the dataset:

   data.isnull().sum()

   This command counts the number of null values in each column. For the dataset in use, there should not be any missing values, as can be seen here:

   

   Figure 2.27: Missing values count

6. Use three standard deviations as the measure to detect any outliers for all the features in the dataset:

   outliers = {}
   for i in range(data.shape[1]):
       min_t = data[data.columns[i]].mean() \
               - (3 * data[data.columns[i]].std())
       max_t = data[data.columns[i]].mean() \
               + (3 * data[data.columns[i]].std())
       count = 0
       for j in data[data.columns[i]]:
           if j < min_t or j > max_t:
               count += 1
       percentage = count / data.shape[0]
       outliers[data.columns[i]] = "%.3f" % percentage
   outliers

   The preceding code snippet performs a for loop through the columns in the dataset in order to evaluate the presence of outliers in each of them. It continues to calculate the minimum and maximum thresholds so that it can count the number of instances that fall outside the range between the thresholds.

   Finally, it calculates the percentage of outliers (that is, the number of outliers divided by the total number of instances) in order to output a dictionary that displays this percentage for each column.

   By printing the resulting dictionary (outliers), it is possible to display a list of all the features (columns) in the dataset, along with the percentage of outliers. According to the result, it is possible to conclude that there is no need to deal with the outlier values, considering that they account for less than 5% of the data, as can be seen in the following screenshot:

   Note

   Note that Jupyter Notebooks can print the value of a variable without the need for the print function whenever the variable is placed at the end of a cell in the notebook. In any other programming platform or any other scenario, make sure to use the print function.

   For instance, an equivalent way (and the best practice) to print the resulting dictionary containing the outliers would be to use the print statement, as follows: print(outliers). This way, the code will have the same output when run in a different programming platform.

Figure 2.28: Outlier participation in each feature

Note

To access the source code for this specific section, please refer to https://packt.live/2CYEglp.

You can also run this example online at https://packt.live/3ePAg4G. You must execute the entire Notebook in order to get the desired result.

You have successfully explored the dataset and dealt with potential issues.

Data Rescaling

Although data does not need to be rescaled to be fed to an algorithm for training, it is an important step if you wish to improve a model's accuracy. This is basically because having different scales for each feature may result in the model assuming that a given feature is more important than others as it has higher numerical values.

Take, for instance, two features, one measuring the number of children a person has and another stating the age of the person. Even though the age feature may have higher numerical values, in a study for recommending schools, the number of children feature may be more important.

Considering this, if all the features are scaled equally, the model can actually give higher weights to those features that matter the most in respect to the target feature, and not the numerical values that they have. Moreover, it can also help accelerate the training process by removing the need for the model to learn from the invariance of the data.

There are two main rescaling methodologies that are popular among data scientists, and although there is no rule for selecting one or the other, it is important to highlight that they are to be used individually (one or the other).

A brief explanation of both of these methodologies can be found here:

• Normalization: This consists of rescaling the values so that all the values of all the features are between zero and one. This is done using the following equation:

Figure 2.29: Data normalization

• Standardization: In contrast, this rescaling methodology converts all the values so that their mean is 0 and their standard deviation is equal to 1. This is done using the following equation:

Figure 2.30: Data standardization

Exercise 2.03: Rescaling Data

In this exercise, we will rescale the data from the previous exercise. Perform the following steps to do so:

Note

Use the same Jupyter notebook that you used in the previous exercise.

1. Separate the features from the target. We are only doing this to rescale the features data:

   X = data.iloc[:, 1:]
   Y = data.iloc[:, 0]

   The preceding code snippet takes the data and uses slicing to separate the features from the target.

2. Rescale the features data by using the normalization methodology. Display the head (that is, the top five instances) of the resulting DataFrame to verify the result:

   X = (X - X.min()) / (X.max() - X.min())
   X.head()

   The output should look as follows:

Figure 2.31: Top instances of the normalized Appliances energy prediction dataset

Note

To access the source code for this specific section, please refer to https://packt.live/2ZojumJ.

You can also run this example online at https://packt.live/2NLVgxq. You must execute the entire Notebook in order to get the desired result.

You have successfully rescaled a dataset.

Splitting the Data

The purpose of splitting the dataset into three subsets is so that the model can be trained, fine-tuned, and measured appropriately, without the introduction of bias. Here is an explanation of each set:

• Training set: As its name suggests, this set is fed to the neural network to be trained. For supervised learning, it consists of the features and the target values. This is typically the largest set out of the three, considering that neural networks require large amounts of data to be trained, as we mentioned previously.
• Validation set (dev set): This set is used mainly to measure the performance of the model in order to make adjustments to the hyperparameters to improve performance. This fine-tuning process is done so that we can configure the hyperparameters that achieve the best results.

  Although the model is not trained on this data, it indirectly has an effect on it, which is why the final measure of performance should not be done on it as it may be a biased measure.

• Testing set: This set does not have an effect on the model, which is why it is used to perform a final evaluation of the model on unseen data, which becomes a guideline of how well the model will perform on future datasets.

There is no actual science on the perfect ratio for splitting data into the three sets mentioned, considering that every data problem is different and developing deep learning solutions usually requires a trial-and-error methodology. Nevertheless, it is widely known that larger datasets (hundreds of thousands and millions of instances) should have a split ratio of 98:1:1 for each set, considering that it is crucial to use as much data as possible for the training set. For a smaller dataset, the conventional split ratio is 60:20:20.

Exercise 2.04: Splitting a Dataset

In this exercise, we will split the dataset from the previous exercise into three subsets. For the purpose of learning, we will explore two different approaches. First, the dataset will be split using indexing. Next, scikit-learn's train_test_split() function will be used for the same purpose, thereby achieving the same result with both approaches. Perform the following steps to complete this exercise:

Note

Use the same Jupyter notebook that you used in the previous exercise.

1. Print the shape of the dataset in order to determine the split ratio to be used:

   X.shape

   The output from this operation should be (19735, 27). This means that it is possible to use a split ratio of 60:20:20 for the training, validation, and test sets.

2. Get the value that you will use as the upper bound of the training and validation sets. This will be used to split the dataset using indexing:

   train_end = int(len(X) * 0.6)
   dev_end = int(len(X) * 0.8)

   The preceding code determines the index of the instances that will be used to divide the dataset through slicing.

3. Shuffle the dataset:

   X_shuffle = X.sample(frac=1, random_state=0)
   Y_shuffle = Y.sample(frac=1, random_state=0)

   Using the pandas sample function, it is possible to shuffle the elements in the features and target matrices. By setting frac to 1, we ensure that all the instances are shuffled and returned in the output from the function. Using the random_state argument, we ensure that both datasets are shuffled equally.

4. Use indexing to split the shuffled dataset into the three sets for both the features and the target data:

   x_train = X_shuffle.iloc[:train_end,:]
   y_train = Y_shuffle.iloc[:train_end]
   x_dev = X_shuffle.iloc[train_end:dev_end,:]
   y_dev = Y_shuffle.iloc[train_end:dev_end]
   x_test = X_shuffle.iloc[dev_end:,:]
   y_test = Y_shuffle.iloc[dev_end:]

5. Print the shapes of all three sets:

   print(x_train.shape, y_train.shape)
   print(x_dev.shape, y_dev.shape)
   print(x_test.shape, y_test.shape)

   The result of the preceding operation should be as follows:

   (11841, 27) (11841,)
   (3947, 27) (3947,)
   (3947, 27) (3947,)

6. Import the train_test_split() function from scikit-learn's model_selection module:

   from sklearn.model_selection import train_test_split

   Note

   Although the different packages and libraries are being imported as they are needed for practical learning purposes, it is always good practice to import them at the beginning of your code.

7. Split the shuffled dataset:

   x_new, x_test_2, \
   y_new, y_test_2 = train_test_split(X_shuffle, Y_shuffle, \
                                      test_size=0.2, \
                                      random_state=0)
   dev_per = x_test_2.shape[0]/x_new.shape[0]
   x_train_2, x_dev_2, \
   y_train_2, y_dev_2 = train_test_split(x_new, y_new, \
                                         test_size=dev_per, \
                                         random_state=0)

   The first line of code performs an initial split. The function takes the following as arguments:

   X_shuffle, Y_shuffle: The datasets to be split, that is, the features dataset, as well as the target dataset (also known as X and Y)

   test_size: The percentage of instances to be contained in the testing set

   random_state: Used to ensure the reproducibility of the results

   The result from this line of code is the division of each of the datasets (X and Y) into two subsets.

   To create an additional set (the validation set), we will perform a second split. The second line of the preceding code is in charge of determining the test_size to be used for the second split so that both the testing and validation sets have the same shape.

   Finally, the last line of code performs the second split using the value that was calculated previously as the test_size.

8. Print the shape of all three sets:

   print(x_train_2.shape, y_train_2.shape)
   print(x_dev_2.shape, y_dev_2.shape)
   print(x_test_2.shape, y_test_2.shape)

   The result from the preceding operation should be as follows:

   (11841, 27) (11841,)
   (3947, 27) (3947,)
   (3947, 27) (3947,)

   As we can see, the resulting sets from both approaches have the same shapes. Using one approach or the other is a matter of preference.

   Note

   To access the source code for this specific section, please refer to https://packt.live/2VxvroW.

   You can also run this example online at https://packt.live/3gcm5H8. You must execute the entire Notebook in order to get the desired result.

You have successfully split the dataset into three subsets.

Disadvantages of Failing to Prepare Your Data

Although the process of preparing the dataset is time-consuming and may be tiring when dealing with large datasets, the disadvantages of failing to do so are even more inconvenient:

• Longer training times: Data containing noise, missing values, and redundant or irrelevant columns takes considerably longer to train and, in most cases, this delay in time is even longer than the time it takes to prepare the data. For instance, during data preparation, it may be determined that five columns are irrelevant for the purpose of the study, which may reduce the dataset considerably, and hence reduce the training times considerably.
• Introduction of bias: Uncleaned data usually contains errors or missing values that can deviate the model from the truth. For instance, missing values can cause the model to make inferences that are not true, which, in turn, creates a model that does not represent the data.
• Avoid generalization: Outliers and noisy values prevent the model from making generalizations of the data, which is crucial for building a model that represents the current training data, as well as future unseen data. For example, a dataset containing a variable for age that contains entries of people who are over 100 years old may result in a model that accounts for those users who, in reality, represent a very small portion of the population.

Activity 2.01: Performing Data Preparation

In this activity, we will prepare a dataset containing a list of songs, each with several attributes that help determine the year they were released. This data preparation step is crucial for the next activity in this chapter. Let's look at the following scenario.

You work at a music record company and your boss wants to uncover the details that characterize records from different time periods, which is why they have put together a dataset that contains data on 515,345 records, with release years ranging from 1922 to 2011. They have tasked you with preparing the dataset so that it is ready to be fed to a neural network. Perform the following steps to complete this activity:

Note

To download the dataset for this activity, visit the following UC Irvine Machine Learning Repository URL: https://archive.ics.uci.edu/ml/datasets/YearPredictionMSD.

Citation: Dua, D. and Graff, C. (2019). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science.

It is also available at this book's GitHub repository: https://packt.live/38kZzZR

1. Import the required libraries.
2. Using pandas, load the .csv file.
3. Verify whether any qualitative data is present in the dataset.
4. Check for missing values.

   You can also add an additional sum() function to get the sum of missing values in the entire dataset, without discriminating by column.

5. Check for outliers.
6. Separate the features from the target data.
7. Rescale the data using the standardization methodology.
8. Split the data into three sets: training, validation, and test. Use whichever approach you prefer.

   Note

   The solution for this activity can be found via this link.

Building a neural network, in general terms, can be achieved either on a very simple level using libraries such as scikit-learn (not suitable for deep learning), which perform all the math for you without much flexibility, or on a very complex level by coding every single step of the training process from scratch, or by using a more robust framework, which allows great flexibility.

PyTorch was built considering the input of many developers in the field and has the advantage of allowing both approximations in the same place. As we mentioned previously, it has a neural network module that was built to allow easy predefined implementations of simple architectures using the sequential container, while at the same time allowing for the creation of custom modules that introduce flexibility to the process of building very complex architectures.

In this section, we will discuss the use of the sequential container for developing deep neural networks in order to demystify their complexity. Nevertheless, in later sections of this book, we will move on and explore more complex and abstract applications, which can also be achieved with very little effort.

As we mentioned previously, the sequential container is a module that was built to contain sequences of modules that follow an order. Each of the modules it contains will apply some computation to a given input to arrive at an outcome.

Some of the most popular modules (layers) that can be used inside the sequential container to develop regular classification models are explained here:

Note

The modules that are used for other types of architectures, such as CNNs and RNNs, will be explained in subsequent chapters.

• Linear layer: This applies a linear transformation to the input data while keeping internal tensors to hold the weights and biases. It receives the size of the input sample (the number of features of the dataset or the number of outputs from the previous layer), the size of the output sample (the number of units in the current layer, which will be the number of outputs), and whether to use a tensor of biases during the training process (which is set to True by default) as arguments.
• Activation functions: They receive the output from the linear layer as input in order to break the linearity. There are several activation functions, as explained previously, that can be added to the sequential container. The most commonly used ones are explained here:

  ReLU: This applies the rectified linear unit function to the tensor containing the input data. The only argument it takes in is whether the operation should be done in-situ, which is set to False by default.

  Tanh: This applies the element-wise tanh function to the tensor containing the input data. It does not take any arguments.

  Sigmoid: This applies the previously explained sigmoid function to the tensor containing the input data. It does not take any arguments.

  Softmax: This applies the softmax function to an n-dimensional tensor containing the input data. The output is rescaled so that the elements of the tensor lie in a range between zero and one, and sum to one. It takes the dimension along which the softmax function should be computed as an argument.

• Dropout layer: This module randomly zeroes some of the elements of the input tensor, according to a set probability. It takes the probability to use for the random selection, as well as whether the operation should be done in-situ, which is set to False by default, as arguments. This technique is commonly used for dealing with overfitted models, which will be explained in more detail later.
• Normalization layer: There are different methodologies that can be used to add a normalization layer in the sequential container. Some of them include BatchNorm1d, BatchNorm2d, and BatchNorm3d. The idea behind this is to normalize the output from the previous layer, which ultimately achieves similar accuracy levels at lower training times.

Exercise 2.05: Building a Deep Neural Network Using PyTorch

In this exercise, we will use the PyTorch library to define the architecture of a deep neural network of four layers, which will then be trained with the dataset we prepared in the previous exercises. Perform the following steps to do so:

Note

Use the same Jupyter notebook that you used in the previous exercise.

1. Import the PyTorch library, called torch, as well as the nn module from PyTorch:

   import torch
   import torch.nn as nn

   Note

   torch.manual_seed(0) is used in this exercise in order to ensure the reproducibility of the results obtained in this book's GitHub repository.

2. Separate the feature columns from the target for each of the sets we created in the previous exercise. Additionally, convert the final DataFrames into tensors:

   x_train = torch.tensor(x_train.values).float()
   y_train = torch.tensor(y_train.values).float()
   x_dev = torch.tensor(x_dev.values).float()
   y_dev = torch.tensor(y_dev.values).float()
   x_test = torch.tensor(x_test.values).float()
   y_test = torch.tensor(y_test.values).float()

3. Define the network architecture using the sequential() container. Make sure to create a four-layer network. Use ReLU activation functions for the first three layers and leave the last layer without an activation function, considering the fact that we are dealing with a regression problem.

   The number of units for each layer should be 100, 50, 25, and 1:

   model = nn.Sequential(nn.Linear(x_train.shape[1], 100), \
                         nn.ReLU(), \
                         nn.Linear(100, 50), \
                         nn.ReLU(), \
                         nn.Linear(50, 25), \
                         nn.ReLU(), \
                         nn.Linear(25, 1))

4. Define the loss function as the MSE:

   loss_function = torch.nn.MSELoss()

5. Define the optimizer algorithm as the Adam optimizer:

   optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

6. Use a for loop to train the network over the training data for 1,000 iteration steps:

   for i in range(1000):
       y_pred = model(x_train).squeeze()
       loss = loss_function(y_pred, y_train)
       optimizer.zero_grad()
       loss.backward()
       optimizer.step()
       if i%100 == 0:
           print(i, loss.item())

   Note

   The squeeze() function is used to remove the additional dimension of y_pred, which is converted from being of size [3000,1] to [3000].

   This is crucial considering that y_train is one-dimensional and both tensors need to have the same dimensions to be fed to the loss function.

   Running the preceding snippet will yield an output similar to the following:

   

   Figure 2.32: Loss value for different iteration steps

   As can be seen, the loss value continually decreases over time.

7. To test the model, perform a prediction on the first instance of the testing set and compare it with the ground truth (target value):

   pred = model(x_test[0])
   print("Ground truth:", y_test[0].item(), \
         "Prediction:",pred.item())

   The output should look similar to the following:

   Ground truth: 60.0 Prediction: 69.5818099975586

   As you can see, the ground truth value (60) is fairly close to the predicted one (69.58).

   Note

   To access the source code for this specific section, please refer to https://packt.live/2NJsQUz.

   You can also run this example online at https://packt.live/38nrnNh. You must execute the entire Notebook in order to get the desired result.

You have successfully created and trained a deep neural network to solve a regression problem.

Activity 2.02: Developing a Deep Learning Solution for a Regression Problem

In this activity, we will create and train a neural network to solve the regression problem we mentioned in the previous activity. Let's look at the scenario.

You continue to work at the music record company and, after seeing the great job you did preparing the dataset, your boss has trusted you with the task of defining the network's architecture, as well as training it with the prepared dataset. Perform the following steps to complete this activity:

Note

Use the same Jupyter notebook that you used in the previous activity.

1. Import the required libraries.
2. Split the features from the targets for all three sets of data that we created in the previous activity. Convert the DataFrames into tensors.
3. Define the architecture of the network. Feel free to try different combinations of the number of layers and the number of units per layer.
4. Define the loss function and the optimizer algorithm.
5. Use a for loop to train the network for 3,000 iteration steps.
6. Test your model by performing a prediction on the first instance of the test set and comparing it with the ground truth.

Your output should look similar to this:

Ground truth: 1995.0 Prediction: 1998.0279541015625

Note

The solution for this activity can be found via this link.

The theory that gave birth to neural networks was developed decades ago by Frank Rosenblatt. It started with the definition of the perceptron, a unit inspired by the human neuron, that takes data as input to perform a transformation on it. The theory behind the perceptron consisted of assigning weights to input data to perform a calculation so that the end result would be either one thing or the other, depending on the outcome.

The most widely known form of neural networks is the one that's created from a succession of perceptrons, stacked together in layers, where the output from one column of perceptrons (layer) is the input for the following one.

The typical learning process for a neural network was explained. Here, there are three main processes to consider: forward propagation, the calculation of the loss function, and backpropagation.

The end goal of this procedure is to minimize the loss function by updating the weights and biases that accompany each of the input values in every neuron of the network. This is achieved through an iterative process that can take minutes, hours, or even weeks, depending on the nature of the data problem.

The main architecture of the three main types of neural networks was also discussed: the artificial neural network, the convolutional neural network, and the recurrent neural network. The first is used to solve traditional classification or regression problems, the second one is widely popular for its capacity to solve computer vision problems (for instance, image classification), and the third one is capable of processing data in sequence, which is useful for tasks such as language translation.

In the next chapter, the main differences between solving regression and a classification data problem will be discussed. You will also learn how to solve a classification data problem, as well as how to improve its performance and how to deploy the model.