Deep Autoencoders

This chapter introduces the concept of deep belief networks and the significance of this type of deep unsupervised learning. It explains such concepts by introducing deep autoencoders along with two regularization techniques that can help create robust models. These regularization techniques, batch normalization and dropout, have been known to facilitate the learning of deep models and have been widely adopted. We will demonstrate the power of a deep autoencoder on MNIST and on a much harder dataset known as CIFAR-10, which contains color images.

By the end of this chapter, you will appreciate the benefits of making deep belief networks by observing the ease of modeling and quality of the output that they provide. You will be able to implement your own deep autoencoder and prove to yourself that deeper models are better than shallow models for most tasks. You...

Introducing deep belief networks

In machine learning, there is a field that is often discussed when talking about deep learning (DL), called deep belief networks (DBNs) (Sutskever, I., and Hinton, G. E. (2008)). Generally speaking, this term is used also for a type of machine learning model based on graphs, such as the well-known Restricted Boltzmann Machine. However, DBNs are usually regarded as part of the DL family, with deep autoencoders as one of the most notable members of that family.

Deep autoencoders are considered DBNs in the sense that there are latent variables that are only visible to single layers in the forward direction. These layers are usually many in number compared to autoencoders with a single pair of layers. One of the main tenets of DL and DBNs in general is that during the learning process, there is different knowledge represented across different sets of layers. This knowledge representation is learned by feature learning without a bias toward a specific class...

Making deep autoencoders

An autoencoder can be called deep so long as it has more than one pair of layers (an encoding one and a decoding one). Stacking layers on top of each other in an autoencoder is a good strategy to improve its power for feature learning in finding unique latent spaces that can be highly discriminatory in classification or regression applications. However, in Chapter 7, Autoencoders, we covered how to stack layers onto an autoencoder, and we will do that again, but this time we will use a couple of new types of layers that are beyond the dense layers we have been using. These are the batch normalization and dropout layers.

There are no neurons in these layers; however, they act as mechanisms that have very specific purposes during the learning process that can lead to more successful outcomes by means of preventing overfitting or reducing numerical instabilities. Let's talk about each of these and then we will continue to experiment with both of these on a...

Exploring latent spaces with deep autoencoders

Latent spaces, as we defined them in Chapter 7, Autoencoders, are very important in DL because they can lead to powerful decision-making systems that are based on assumed rich latent representations. And, once again, what makes the latent spaces produced by autoencoders (and other unsupervised models) rich in their representations is that they are not biased toward particular labels.

In Chapter 7, Autoencoders, we explored the MNIST dataset, which is a standard dataset in DL, and showed that we can easily find very good latent representations with as few as four dense layers in the encoder and eight layers for the entire autoencoder model. In the next section, we will take on a much more difficult dataset known as CIFAR-10, and then we will come back to explore the latent representation of the IMDB dataset, which we have already explored briefly in the previous sections of this chapter.

CIFAR-10

In 2009, the Canadian Institute for Advanced...

Summary

This intermediate chapter showed the power of deep autoencoders when combined with regularization strategies such as dropout and batch normalization. We implemented an autoencoder that has more than 30 layers! That's deep! We saw that in difficult problems a deep autoencoder can offer an unbiased latent representation of highly complex data, as most deep belief networks do. We looked at how dropout can reduce the risk of overfitting by ignoring (disconnecting) a fraction of the neurons at random in every learning step. Furthermore, we learned that batch normalization can offer stability to the learning algorithm by gradually adjusting the response of some neurons so that activation functions and other connected neurons don't saturate or overflow numerically.

At this point, you should feel confident applying batch normalization and dropout strategies in a deep autoencoder model. You should be able to create your own deep autoencoders and apply them to different tasks...

Questions and answers

1. Which regularization strategy discussed in this chapter alleviates overfitting in deep models?

Dropout.

2. Does adding a batch normalization layer make the learning algorithm have to learn more parameters?

Actually, no. For every layer in which dropout is used, there will be only two parameters for every neuron to learn: . If you do the math, the addition of new parameters is rather small.

3. What other deep belief networks are out there?

Restricted Boltzmann machines, for example, are another very popular example of deep belief networks. Chapter 10, Restricted Boltzmann Machines, will cover these in more detail.

4. How come deep autoencoders perform better on MNIST than on CIFAR-10?

Actually, we do not have an objective way of saying that deep autoencoders are better on these datasets. We are biased in thinking about it in terms of clustering and data labels. Our bias in thinking about the latent representations in Figure 8.12 and Figure 8.16 in terms of labels...

References

• Sutskever, I., & Hinton, G. E. (2008). Deep, narrow sigmoid belief networks are universal approximators. Neural computation, 20(11), 2629-2636.
• Sainath, T. N., Kingsbury, B., & Ramabhadran, B. (2012, March). Auto-encoder bottleneck features using deep belief networks. In 2012 IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 4153-4156). IEEE.
• Wu, K., & Magdon-Ismail, M. (2016). Node-by-node greedy deep learning for interpretable features. arXiv preprint arXiv:1602.06183.
• Ioffe, S., & Szegedy, C. (2015, June). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In International Conference on Machine Learning (ICML) (pp. 448-456).
• Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1), 1929-1958.
• Duchi, J., Hazan, E., & Singer, Y...