Chapter 6.  Autoencoders

	"People worry that computers will get too smart and take over the world, but the real problem is that they're too stupid and they've already taken over the world."
	--Pedro Domingos

In the last chapter, we discussed a generative model called Restricted Boltzmann machine. In this chapter, we will introduce one more generative model called autoencoder. Autoencoder, a type of artificial neural network, is generally used for dimensionality reduction, feature learning, or extraction.

As we move on with this chapter, we will discuss the concept of autoencoder and its various forms in detail. We will also explain the terms regularized autoencoder and sparse autoencoder. The concept of sparse coding, and selection criteria of the sparse factor in a sparse autoencoder will be taken up. Later, we will talk about the deep learning model, deep autoencoder, and its implementation using Deeplearning4j. Denoising autoencoder is one more form of a traditional autoencoder, which...

Autoencoder

An autoencoder is a neural network with one hidden layer, which is trained to learn an identity function that attempts to reconstruct its input to its output. In other words, the autoencoder tries to copy the input data by projecting onto a lower dimensional subspace defined by the hidden nodes. The hidden layer, h, describes a code, which is used to represent the input data and its structure. This hidden layer is thus forced to learn the structure from its input training dataset so that it can copy the input at the output layer.

The network of an autoencoder can be split into two parts: encoder and decoder. The encoder is described by the function h=f (k), and a decoder that tries to reconstruct or copy is defined by r = g (h). The basic idea of autoencoder should be to copy only those aspects of the inputs which are prioritized, and not to create an exact replica of the input. They are designed in such a way so as to restrict the hidden layer to copy only approximately, and...

Sparse autoencoders

Distributed sparse representation is one of the primary keys to learn useful features in deep learning algorithms. Not only is it a coherent mode of data representation, but it also helps to capture the generation process of most of the real world dataset. In this section, we will explain how autoencoders encourage sparsity of data. We will start with introducing sparse coding. A code is termed as sparse when an input provokes the activation of a relatively small number of nodes of a neural network, which combine to represent it in a sparse way. In deep learning technology, a similar constraint is used to generate the sparse code models to implement regular autoencoders, which are trained with sparsity constants called sparse autoencoders.

Sparse coding

Sparse coding is a type of unsupervised method to learn sets of overcomplete bases in order to represent the data in a coherent and efficient way. The primary goal of sparse coding is to determine a set of vectors (n) v_i...

Deep autoencoders

So far, we have talked only about single-layer encoders and single-layer decoders for a simple autoencoder. However, a deep autoencoder with more than one encoder and decoder brings more advantages.

Feed-forward networks perform better when they are deep. Autoencoders are basically feed-forward networks; hence, the advantages of a basic feed-forward network can also be applied to autoencoders. The encoders and decoders are autoencoders, which also work like a feed-forward network. Hence, we can deploy the advantages of the depth of a feed-forward network in these components also.

In this context, we can also talk about the universal approximator theorem, which ensures that a feed-forward neural network with at least one hidden layer, and with enough hidden units, can produce an approximation of any arbitrary function to any degree of accuracy. Following this concept, a deep autoencoder having at least one hidden layer, and containing sufficient hidden units, can approximate...

Denoising autoencoder

The reconstruction of output from input does not always guarantee the desired output, and can sometimes end up in simply copying the input. To prevent such a situation, in [134], a different strategy has been proposed. In that proposed architecture, rather than putting some constraints in the representation of the input data, the reconstruction criteria is built, based on cleaning the partially corrupted input.

"A good representation is one that can be obtained robustly from a corrupted input and that will be useful for recovering the corresponding clean input."

A denoising autoencoder is a type of autoencoder which takes corrupted data as input, and the model is trained to predict the original, clean, and uncorrupted data as its output. In this section, we will explain the basic idea behind designing a denoising autoencoder.

Architecture of a Denoising autoencoder

The primary idea behind a denoising autoencoder is to introduce a corruption process, Q (k^/ | k), and reconstruct...

Applications of autoencoders

Autoencoders can be successfully applied in many use cases, and hence, have gained much popularity in the world of deep learning. In this section, we will discuss the important applications and uses of autoencoders:

• Dimensionality reduction: If you remember, in Chapter 1, Introduction to Deep Learning, we introduced the concept of the 'curse of dimensionality'. Dimensionality reduction was one of the first applications of deep learning. Autoencoders were initially studied to overcome the issues with the curse of dimensionality. We have already got a fair idea from this chapter how deep autoencoders work on higher-dimensional data to reduce the dimensionality in the final output.

• Information Retrieval: One more important application of autoencoders is in information retrieval. Information retrieval basically means to search for some entries, which match with an entered query, in a database. Searching in high-dimensional data is generally a cumbersome task; however...

Summary

Autoencoders, one of the most popular and widely applicable generative models, have been discussed in this chapter. Autoencoders basically help two phases: one is the encoder phase and the other is the decoder phase. In this chapter, we elaborated on both of these phases with suitable mathematical explanations. Going forward, we explained a special kind of autoencoder called the sparse autoencoder. We also discussed how autoencoders can be used in the world of deep neural networks by explaining deep autoencoders. Deep autoencoders consist of layers of Restricted Boltzmann machines, which take part in the encoder and decoder phases of the network. We explained how to deploy deep autoencoders using Deeplearning4j, by loading chunks of the input dataset into a Hadoop Distributed File System. Later in this chapter, we introduced the most popular form of autoencoder called the denoising autoencoder and its deep network version known as the stacked denoising autoencoder. The implementation...