Autoencoders

Autoencoders are neural networks that learn by unsupervised learning, also sometimes called semi-supervised learning, since the input is treated as the target too. In this chapter, you will learn about and implement different variants of autoencoders and eventually learn how to stack autoencoders. We will also see how autoencoders can be used to create MNIST digits, and finally, also cover the steps involved in building a long short-term memory autoencoder to generate sentence vectors. This chapter includes the following topics:

• Vanilla autoencoders
• Sparse autoencoders
• Denoising autoencoders
• Convolutional autoencoders
• Stacked autoencoders
• Generating sentences using LSTM autoencoders
• Variational autoencoders for generating images

All the code files for this chapter can be found at https://packt.link/dltfchp8

Let’s begin!

Introduction to autoencoders

Autoencoders are a class of neural networks that attempt to recreate input as their target using backpropagation. An autoencoder consists of two parts: an encoder and a decoder. The encoder will read the input and compress it to a compact representation, and the decoder will read the compact representation and recreate the input from it. In other words, the autoencoder tries to learn the identity function by minimizing the reconstruction error.

They have an inherent capability to learn a compact representation of data. They are at the center of deep belief networks and find applications in image reconstruction, clustering, machine translation, and much more.

You might think that implementing an identity function using deep neural networks is boring; however, the way in which this is done makes it interesting. The number of hidden units in the autoencoder is typically fewer than the number of input (and output) units. This forces the encoder to...

Vanilla autoencoders

The vanilla autoencoder, as proposed by Hinton in his 2006 paper Reducing the Dimensionality of Data with Neural Networks, consists of one hidden layer only. The number of neurons in the hidden layer is fewer than the number of neurons in the input (or output) layer.

This results in producing a bottleneck effect in the flow of information in the network. The hidden layer (y) between the encoder input and decoder output is also called the “bottleneck layer.” Learning in the autoencoder consists of developing a compact representation of the input signal at the hidden layer so that the output layer can faithfully reproduce the original input.

In Figure 8.2, you can see the architecture of a vanilla autoencoder:

Figure 8.2: Architecture of the vanilla autoencoder

Let’s try to build a vanilla autoencoder. While in the paper Hinton used it for dimension reduction, in the code to follow, we will use autoencoders for image reconstruction...

Sparse autoencoder

The autoencoder we covered in the previous section works more like an identity network; it simply reconstructs the input. The emphasis is on reconstructing the image at the pixel level, and the only constraint is the number of units in the bottleneck layer. While it is interesting, pixel-level reconstruction is primarily a compression mechanism and does not necessarily ensure that the network will learn abstract features from the dataset. We can ensure that a network learns abstract features from the dataset by adding further constraints.

In sparse autoencoders, a sparse penalty term is added to the reconstruction error. This tries to ensure that fewer units in the bottleneck layer will fire at any given time. We can include the sparse penalty within the encoder layer itself.

In the following code, you can see that the dense layer of Encoder now has an additional parameter, activity_regularizer:

class SparseEncoder(K.layers.Layer):
    def __init__...

Denoising autoencoders

The two autoencoders that we have covered in the previous sections are examples of undercomplete autoencoders, because the hidden layer in them has lower dimensionality compared to the input (output) layer. Denoising autoencoders belong to the class of overcomplete autoencoders because they work better when the dimensions of the hidden layer are more than the input layer.

A denoising autoencoder learns from a corrupted (noisy) input; it feeds its encoder network the noisy input, and then the reconstructed image from the decoder is compared with the original input. The idea is that this will help the network learn how to denoise an input. It will no longer just make pixel-wise comparisons, but in order to denoise, it will learn the information of neighboring pixels as well.

A denoising autoencoder has two main differences from other autoencoders: first, n_hidden, the number of hidden units in the bottleneck layer is greater than the number of units in...

Stacked autoencoder

Until now, we have restricted ourselves to autoencoders with only one hidden layer. We can build deep autoencoders by stacking many layers of both encoders and decoders; such an autoencoder is called a stacked autoencoder. The features extracted by one encoder are passed on to the next encoder as input. The stacked autoencoder can be trained as a whole network with the aim of minimizing the reconstruction error. Alternatively, each individual encoder/decoder network can first be pretrained using the unsupervised method you learned earlier, and then the complete network can be fine-tuned. When the deep autoencoder network is a convolutional network, we call it a convolutional autoencoder. Let us implement a convolutional autoencoder in TensorFlow next.

Convolutional autoencoder for removing noise from images

In the previous section, we reconstructed handwritten digits from noisy input images. We used a fully connected network as the encoder and decoder for...

Variational autoencoders

Like DBNs (Chapter 7, Unsupervised Learning) and GANs (see Chapter 9, Generative Models, for more details), variational autoencoders are also generative models. Variational autoencoders (VAEs) are a mix of the best neural networks and Bayesian inference. They are one of the most interesting neural networks and have emerged as one of the most popular approaches to unsupervised learning. They are autoencoders with a twist. Along with the conventional encoder and decoder network of autoencoders, they have additional stochastic layers. The stochastic layer, after the encoder network, samples the data using a Gaussian distribution, and the one after the decoder network samples the data using Bernoulli’s distribution. Like GANs, VAEs can be used to generate images and figures based on the distribution they have been trained on.

VAEs allow one to set complex priors in the latent space and thus learn powerful latent representations. Figure 8.14 describes...

Summary

In this chapter, we’ve had an extensive look at a new generation of deep learning models: autoencoders. We started with the vanilla autoencoder, and then moved on to its variants: sparse autoencoders, denoising autoencoders, stacked autoencoders, and convolutional autoencoders. We used the autoencoders to reconstruct images, and we also demonstrated how they can be used to clean noise from an image. Finally, the chapter demonstrated how autoencoders can be used to generate sentence vectors and images. The autoencoders learned through unsupervised learning.

In the next chapter, we will delve deeper into generative adversarial networks, another interesting deep learning model that learns via an unsupervised learning paradigm.

References

1. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1985). Learning Internal Representations by Error Propagation. No. ICS-8506. University of California, San Diego. La Jolla Institute for Cognitive Science: http://www.cs.toronto.edu/~fritz/absps/pdp8.pdf
2. Hinton, G. E. and Salakhutdinov, R. R. (2016). Reducing the dimensionality of data with neural networks. science 313.5786: 504–507: https://www.cs.toronto.edu/~hinton/science.pdf
3. Masci, J. et al. (2011). Stacked convolutional auto-encoders for hierarchical feature extraction. Artificial Neural Networks and Machine Learning–ICANN 2011: 52–59: https://www.semanticscholar.org/paper/Reducing-the-dimensionality-of-data-with-neural-Hinton-Salakhutdinov/46eb79e5eec8a4e2b2f5652b66441e8a4c921c3e
4. Japkowicz, N., Myers, C., and Gluck, M. (1995). A novelty detection approach to classification. IJCAI. Vol: https://www.ijcai.org/Proceedings/95-1/Papers/068.pdf
5. Sedhain, S. (2015)...