Architecture Optimization of Deep Learning Networks

This chapter describes how genetic algorithms can be used to improve the performance of artificial neural network-based models by optimizing the network architecture of these models. We will start with a brief introduction to neural networks and deep learning. After introducing the Iris dataset and the Multilayer Perceptron classifier, we will demonstrate network architecture optimization using a genetic algorithm-based solution. Then, we will extend this approach to combine network architecture optimization with model hyperparameter tuning, which will be jointly carried out by a genetic algorithm-based solution.

In this chapter, we will cover the following topics:

• Understanding the basic concepts of artificial neural networks and deep learning
• The Iris dataset and the Multilayer Perceptron (MLP) classifier
• Enhancing the performance...

Technical requirements

In this chapter, we will be using Python 3 with the following supporting libraries:

• deap
• numpy
• sklearn

In addition, we will be using the UCI Iris flower dataset (https://archive.ics.uci.edu/ml/datasets/Iris).

The programs that will be used in this chapter can be found in this book's GitHub repository at https://github.com/PacktPublishing/Hands-On-Genetic-Algorithms-with-Python/tree/master/Chapter09.

Check out the following video to see the Code in Action:
http://bit.ly/317KCXA

Artificial neural networks and deep learning

Neural networks are among the most commonly used models in machine learning and were inspired by the structure of the human brain. The basic building blocks of these networks are nodes, or neurons, which are based on the biological neuron cell, as depicted in the following diagram:

Biological neuron model
Source: https://pixabay.com/vectors/neuron-nerve-cell-axon-dendrite-296581/

The neuron cell's dendrites, which are surrounding the cell body on the left-hand side of the preceding diagram, are used as inputs from multiple similar cells, while the long axon, coming out of the cell body, serves as output and can be connected to multiple other cells.

This structure is mimicked by the artificial model called the perceptron, illustrated as follows:

Artificial neuron model – the perceptron

The perceptron calculates the output...

Optimizing the architecture of a deep learning classifier

When creating a neural network model so that we can carry out a given machine learning task, one crucial design decision that needs to be made is the configuration of the network architecture. In the case of the Multilayer Perceptron, the number of nodes in the input and output layers is determined by the characteristics of the problem at hand. Therefore, the choices to be made are about the hidden layers – how many layers, and how many nodes in each layer. Some rules of thumb can be employed for making these decisions, but in many cases, identifying the best choices can turn into a cumbersome trial-and-error process.

One way to handle network architecture parameters is to consider them as hyperparameters of the model since they need to be determined before training is done and affect the training's results...

Combining architecture optimization with hyperparameter tuning

While optimizing the network architecture configuration—the hidden layer parameters—we have been using the default parameters of the MLP classifier. However, as we saw in the previous chapter, tuning the various hyperparameters has the potential to increase the classifier's performance. Can we incorporate hyperparameter tuning into our optimization? As you may have guessed, the answer is yes. But first, let's take a look at the hyperparameters we would like to optimize.

The sklearn implementation of the MLP classifier contains numerous tunable hyperparameters. For our demonstration, we will concentrate on the following hyperparameters:

Name	Type	Description	Default value
activation	{'tanh', 'relu', 'logistic'}	Activation function for the hidden layers	&apos...

Summary

In this chapter, you were introduced to the basic concepts of artificial neural networks and deep learning. After getting acquainted with the Iris dataset and the Multilayer Perceptron (MLP) classifier, you were presented with the notion of network architecture optimization. Next, we demonstrated a genetic algorithm-based optimization of network architecture for the MLP classifier. Finally, we were able to combine network architecture optimization with model hyperparameter tuning with the genetic algorithms process and enhance the performance of the classifier even further.

So far, we have concentrated on supervised learning. In the next chapter, we will look into applying genetic algorithms to reinforcement learning, an exciting and fast-developing branch of machine learning.

Further reading

For more information on the topics that we covered in this chapter, please refer to the following resources:

• Python Deep Learning - Second Edition, Gianmario Spacagna, Daniel Slater et al. January 16, 2019
• Neural Network Projects with Python, James Loy, February 28, 2019
• scikit-learn Multilayer Perceptron Classifier: https://scikit-learn.org/stable/modules/neural_networks_supervised.html
• UCI Machine Learning Repository: https://archive.ics.uci.edu/ml/index.php