Neural Networks

In this chapter, we are going to learn about neural networks. We will start with an introduction to neural networks and the installation of the relevant library. We will then discuss perceptrons and how to build a classifier based on them. After that, we'll go deeper and learn about single-layer neural networks and multi-layer neural networks.

Later, we will see how to use neural networks to build a vector quantizer. We will analyze sequential data using recurrent neural networks, and finally we will use neural networks to build an optical character recognition engine. By the end of this chapter, we will have covered:

• An introduction to neural networks
• Building a Perceptron-based classifier
• Constructing a single-layer neural network
• Constructing a multilayer neural network
• Building a vector quantizer
• Analyzing sequential data using recurrent neural networks
• Visualizing characters in an Optical Character Recognition ...

Introduction to neural networks

One of the fundamental premises of AI is to build systems that can perform tasks that would normally require human intelligence. The human brain is amazing at learning new concepts. Why not use the model of the human brain to build a system? A neural network is a model designed to loosely simulate the learning process of the human brain.

Neural networks are designed such that they can identify the underlying patterns in data and learn from them. They can be used for various tasks such as classification, regression, and segmentation. One drawback of neural networks is that we need to convert any given data into a numerical format before feeding it into the neural network. For example, we deal with many different types of data including visual, textual, and time series. We need to figure out how to represent problems in a way that can be understood by neural networks. To understand this process, let's first consider how to build a neural network...

Building a Perceptron-based classifier

Neurons, dendrites, and axons comprise the building blocks of the brain. Similarly, perceptrons are the most basic structure in a neural network.

The evolution of the neural networks has gone through many changes. Their development is grounded on the neurological work performed by Santiago Ramon y Cajal and Sir Charles Scott Sherrington. Ramon y Cajal was a pioneer in the exploration of the structure of nervous tissue and demonstrated that:

• Neurons can communicate with each other
• Neurons are physically separate from other neurons

Leveraging the research of Ramon y Cajal and Sherrington, Warren McCulloch and Walter Pitts in their 1943 paper, A Logical Calculus of Ideas Immanent in Nervous Activity, described an architecture borrowing from the structure of neurons having a binary threshold activation function analogous to first order logic sentences.

The following is a basic representation of McCulloch and...

Constructing a single-layer neural network

Having a model that has a few perceptrons is a good start and it allowed us to get a fundamental understanding of this exciting concept, but to really solve problems, such a simple model will not be sufficient. The human brain has approximately 85 billion neurons. We won't be building a neural network that has that many nodes, but this number gives you an idea of what it takes to solve complicated problems. Before we build a model with billions of nodes, let's take the next step to building a network with a single-layer. This single-layer neural network consists of independent neurons acting on input data to produce an output. Let's get to it.

Create a new Python file and import the following packages:

import numpy as np
import matplotlib.pyplot as plt
import neurolab as nl

We will use the input data from the file data_simple_nn.txt provided to you. Each line in this file contains four numbers. The...

Constructing a multi-layer neural network

So, we enhanced our model from a few nodes to a single-layer, but we are still far away from 85 billion nodes. We won't get to that in this section either, but let's take another step in the right direction. The human brain does not use a single-layer model. The output from some neurons becomes the input for other neurons and so on. A model that has this characteristic is known as a multi-layer neural network. This type of architecture yields higher accuracy, and it enables us to solve more complex and more varied problems. Let's see how we can use NeuroLab to build a multi-layer neural network.

Create a new Python file and import the following packages:

import numpy as np
import matplotlib.pyplot as plt
import neurolab as nl

In the previous two sections, we saw how to use a neural network as a classifier. In this section, we will see how to use a multi-layer neural network as a regressor. Generate...

Building a vector quantizer

Vector Quantization is a quantization technique where the input data is represented by a fixed number of representative points. It is the N-dimensional equivalent of rounding off a number. This technique is commonly used in multiple fields such as voice/image recognition, semantic analysis, and image/voice compression. The history of optimal vector quantization theory goes back to the 1950s in Bell Labs, where research was carried out to optimize signal transmission using discretization procedures. One advantage of vector quantizer neural networks is that they have high interpretability. Let's see how we can build a vector c.

Due to some issues with the current version of NeuroLab (v. 0.3.5), running the following code will throw an error. Fortunately, there is a fix for this, but it involves making a change in the NeuroLab package. Changing line 179 of the net.py file in the NeuroLab package (layer_out.np['w'][n][st:i].fill...

Analyzing sequential data using recurrent neural networks

With all our neural network examples so far, we have been using static data. Neural networks can also be used effectively to build models that process sequential data. Recurrent neural networks (RNNs) are great at modeling sequential data. You can learn more about recurrent neural networks at:

https://www.jeremyjordan.me/introduction-to-recurrent-neural-networks/

When we are working with time series data, we normally cannot use generic learning models. We need to capture the temporal dependencies in the data so that a robust model can be built. Let's see how to build it.

Create a new Python file and import the following packages:

import numpy as np
import matplotlib.pyplot as plt
import neurolab as nl

Define a function to generate the waveforms. Start by defining four sine waves:

def get_data(num_points):
    # Create sine waveforms
    wave_1 = 0.5 * np.sin(np.arange(0, num_points))
   ...

Visualizing characters in an optical character recognition database

Neural networks can also be used for optical character recognition. It is perhaps one of its most common use cases. Converting handwriting to computer characters has been a fundamental problem that many computer scientists have tried to solve and has yet remained elusive. We have made great advances but, for obvious reasons, 100% accuracy will remain out of reach. Why?

Think about this scenario. Have you ever written anything down and five minutes later, you are unable to read your own handwriting? Computers will always have this problem as well. There is an infinite number of ways to write down the number 6 and some of them will look more like a 0 or a 5 than a 6. I might be wrong, but I think we will find a cure for cancer before we are able to find a reliable way for a computer to recognize a doctor's handwriting. We can already achieve high levels of accuracy and the more beautiful the handwriting...

Building an optical character recognition engine

Now that we have learned how to work with this data, let's build an optical character recognition system using neural networks.

Create a new Python file and import the following packages:

import numpy as np
import neurolab as nl

Define the input file:

# Define the input file
input_file = 'letter.data'

Define the number of data points that will be loaded:

# Define the number of datapoints to 
# be loaded from the input file
num_datapoints = 50

Define the string containing all the distinct characters:

# String containing all the distinct characters
orig_labels = 'omandig'

Extract the number of distinct classes:

# Compute the number of distinct characters
num_orig_labels = len(orig_labels)

Define the train and test split. We will use 90% for training and 10% for testing:

# Define the training and testing parameters
num_train = int(0.9 * num_datapoints...

Summary

In this chapter, we learned about neural networks. We discussed how to build and train neural networks. We talked about perceptrons and built a classifier based on that. We learned about single-layer neural networks as well as multi-layer neural networks. We discussed how neural networks could be used to build a vector quantizer. We analyzed sequential data using recurrent neural networks. We then built an optical character recognition engine using neural networks. In the next chapter, we will learn about reinforcement learning and see how to build smart learning agents.