Chapter 12. Learning Features with Unsupervised Generative Networks

This chapter focuses on a new type of model, the generative models, which include Restricted Boltzmann Machines, Deep Belief Networks, Variational Auto Encoders, Autoregressive models, and Generative Adversarial Networks. For the first nets, we've limited the presentation to the theory, while the last is explained in detail with practical code and advice.

These nets do not require any labels to be trained, which is called unsupervised learning. Unsupervised learning helps compute features from the data, without the bias of the labels. These models are generative in the sense that they are trained to generate new data that sounds real.

The following points will be covered:

• Generative models

• Unsupervised learning

• Restricted Boltzmann Machines

• Deep belief networks

• Generative adversarial models

• Semi-supervised learning

Generative models

A generative model in neural processing is a model that generates data given a noise vector z as input:

The purpose of the training is to find the parameters to generate data as close as possible to the real data.

Applications of generative networks include data dimensionality reduction, synthetic data generation, unsupervised feature learning, and pre-training / training efficiency. Pre-training helps generalization because pre-training focuses on the patterns in the data, and less on the data-label relation.

Restricted Boltzmann Machines

A Restricted Boltzmann Machine is the simplest generative net, composed of one fully connected hidden layer, as shown in the picture:

The full Boltzmann Machines have also hidden-to-hidden and visible-to-visible loop connections, while the Restricted version does not have any.

In the general case, RBM are defined as energy-based models, which means that they define a probability distribution through an energy function:

Z is the partition function...

Semi-supervised learning

Last but not least, such generative adversarial networks can be used to enhance supervised learning itself.

Suppose the objective is to classify K classes, for which an amount of labeled data is available. It is possible to add some generated samples to the dataset, which come from a generative model, and consider them as belonging to a (K+1)th class, the fake data class.

Decomposing the training cross-entropy loss of the new classifier between the two sets (real data and fake data) leads to the following formula:

Here, is the probability predicted by the model:

Note that if we know that the data is real:

And training on real data (K classes) would have led to the loss:

Hence the loss of the global classifier can be rewritten:

The second term of the loss corresponds to the standard unsupervised loss for GAN:

The interaction introduced between the supervised and the unsupervised loss is still not well understood but, when the classification is not trivial, an unsupervised...

Further reading

You can refer to the following topics for more insights:

• Deeplearning.net tutorial on RBM: http://deeplearning.net/tutorial/rbm.html

• Deeplearning.net tutorial on Deep Belief Nets: http://deeplearning.net/tutorial/DBN.html

• Deeplearning.net tutorial on generating with RBM-RNN: http://deeplearning.net/tutorial/rnnrbm.html

• Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription, Nicolas Boulanger-Lewandowski, Yoshua Bengio, Pascal Vincent, 2012

• Generative Adversarial Networks, Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio, 2014

• Gans will change the world, Nikolai Yakovenko, 2016 https://medium.com/@Moscow25/

• Pixel Recurrent Neural Networks, Aaron van den Oord, Nal Kalchbrenner, Koray Kavukcuoglu, 2016

• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets, Xi Chen, Yan Duan, Rein Houthooft...

Summary

Generative adversarial networks are a very active area of research today. They belong to the family of generative models, which includes RBM and deep belief networks.

Generative models aim at generating more data, or learning better features for supervised and other tasks in an unsupervised way.

Generative models can be conditioned on some environmental input, and try to find the hidden variables behind the real data.

These models, the most advanced, complete the overview of deep learning nets with Theano. The next chapter will look at some advanced concepts to extend Theano and the future of deep learning.