Autoencoders

This chapter introduces the autoencoder model by explaining the relationship between encoding and decoding layers. We will be showcasing a model that belongs to the unsupervised learning family. This chapter also introduces a loss function commonly associated with the autoencoder model, and it also applies it to the dimensionality reduction of MNIST data and its visualization in an autoencoder-induced latent space.

The following topics will be covered in this chapter:

• Introduction to unsupervised learning
• Encoding and decoding layers
• Applications in dimensionality reduction and visualization
• Ethical implications of unsupervised learning

Introduction to unsupervised learning

As machine learning has progressed over the last few years, I have come across many ways to categorize the different types of learning. Recently, at the NeurIPS 2018 conference in Montreal, Canada, Dr. Alex Graves shared information about the different types of learning, shown in Figure 7.1:

Figure 7.1 – Different types of learning

Such efforts at categorization are very useful today when there are many learning algorithms being studied and improved. The first row depicts active learning, which means that there is a sense of interaction between the learning algorithm and the data. For example, in reinforcement learning and active learning operating over labeled data, the reward policies can inform what type of data the model will read in the following iterations. However, traditional supervised learning, which is what we have studied so far, involves no interaction with the data source and instead assumes that the dataset is fixed and that...

Encoding and decoding layers

The autoencoder can be broken down into two major components that serve specific purposes during an unsupervised learning process. The left side of Figure 7.2 shows an autoencoder that is implemented using fully connected (dense) layers. It receives as input some vector and then it goes into six hidden layers; the first three, with 6, 4, and 2 neurons, respectively, are meant to compress the input data down to two dimensions, since the output of two neurons is two scalar values. This first set of layers is known as the encoder:

Figure 7.2 – Two representations of an autoencoder. Left: full and descriptive model. Right: compact and abstracted model representation

The second set of neurons is meant to reconstruct the input data back to its original dimensionality and values using three layers with 4, 6, and 8 neurons, respectively; this group of layers is known as the decoder.

Note that the last layer of the autoencoder must have the same number of...

Applications in dimensionality reduction and visualization

Among some of the most interesting applications of autoencoders is dimensionality reduction [Wang, Y., et al. (2016)]. Given that we live in a time where data storage is easily accessible and affordable, large amounts of data are currently stored everywhere. However, not everything is relevant information. Consider, for example, a database of video recordings of a home security camera that always faces one direction. Chances are that there is a lot of repeated data in every video frame or image and very little of the data gathered will be useful. We would need a strategy to look at what is really important in those images. Images, by their nature, have a lot of redundant information, and there is usually correlation among image regions, which makes autoencoders very useful in compressing the information in images (Petscharnig, S., et al. (2017)).

To demonstrate the applicability of autoencoders in dimensionality reduction for...

Ethical implications of unsupervised learning

Unsupervised learning, such as what we see happening in the autoencoder we have been exploring so far, is not magical. It is well established and has very rigorous boundaries that are known and pre-defined. It does not have the capability of learning new things outside the limitations given by the data. Remember, unsupervised learning is passive learning as explained in the introductory section of this chapter.

However, even the most robust of unsupervised learning models have ethical risks associated with them. One of the major problems is that they create difficulties when dealing with outliers or data that may contain edge cases. For example, say that there is a large amount of data for IT recruitment, which includes years of experience, current salary, and programming languages that a candidate knows. If the data mostly contains data about candidates with the same programming language experience, and only a few know Python, then those...

Summary

This chapter showed that autoencoders are very simple models that can be used to encode and decode data for different purposes, such as data compression, data visualization, and simply finding latent spaces where only important features are preserved. We showed that the number of neurons and the number of layers in the autoencoder are important for the success of the model. Deeper (more layers) and wider (more neurons) traits are often ingredients for good models, even if that leads to slower training times.

At this point, you should know the difference between supervised and unsupervised learning in terms of passive learning. You should also feel comfortable implementing the two basic components of an autoencoder: the encoder and the decoder. Similarly, you should be able to modify the architecture of an autoencoder to fine-tune it to achieve better performance. Taking the example we discussed in this chapter, you should be able to apply an autoencoder to a dimensionality reduction...

Questions and answers

1. Is overfitting a bad thing for an autoencoder?

Actually, no. You want the autoencoder to overfit! That is, you want it to exactly replicate the input data in the output. However, there is a caveat. Your dataset must be really large in comparison to the size of the model; otherwise, the memorization of the data will prevent the model from generalizing to unseen data.

2. Why did we use two neurons in the encoder's last layer?

For visualization purposes only. The two-dimensional latent space produced by the two neurons allows us to easily visualize the data in the latent space. In the next chapter, we will use other configurations that do not necessarily have a two-dimensional latent space.

3. What is so cool about autoencoders again?

They are simple neural models that learn without a teacher (unsupervised). They are not biased toward learning specific labels (classes). They learn about the world of data through iterative observations, aiming to learn the...

References

• Amari, S. I. (1993). Backpropagation and stochastic gradient descent method. Neurocomputing, 5(4-5), 185-196.
• Wang, Y., Yao, H., & Zhao, S. (2016). Auto-encoder based dimensionality reduction. Neurocomputing, 184, 232-242.
• Petscharnig, S., Lux, M., & Chatzichristofis, S. (2017). Dimensionality reduction for image features using deep learning and autoencoders. In Proceedings of the 15th International Workshop on Content-Based Multimedia Indexing (p. 23). ACM.
• Creswell, A., Arulkumaran, K., & Bharath, A. A. (2017). On denoising autoencoders trained to minimize binary cross-entropy. arXiv preprint arXiv:1708.08487.
• Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.