Chapter 5: Style Transfer

Generative models such as VAE and GAN are great at generating realistic looking images. But we understand very little about the latent variables, let alone how to control them with regard to image generation. Researchers began to explore ways to better represent images aside from pixel distribution. It was found that an image could be disentangled into content and style. Content describes the composition in the image such as a tall building in the middle of the image. On the other hand, style refers to the fine details, such as the brick or stone textures of the wall or the color of the roof. Images showing the same building at different times of the day have different hues and brightness and can be seen as having the same content but different styles.

In this chapter, we will start by implementing some seminal work in neural style transfer to transfer the artistic style of an image. We will then learn to implement feed-forward neural style transfer, which...

Technical requirements

The Jupyter notebooks and codes can be found at the following link:

https://github.com/PacktPublishing/Hands-On-Image-Generation-with-TensorFlow-2.0/tree/master/Chapter05

The notebooks used in the chapter are as follows:

• ch5_neural_style_transfer.ipynb
• ch5_arbitrary_style_transfer.ipynb

Neural style transfer

When convolutional neural networks (CNNs) outperformed all other algorithms in the ImageNet image classification competition, people started to realize the potential of it and began exploring it for other computer vision tasks. In the A Neural Algorithm of Artistic Style paper published in 2015 by Gatys et al., they demonstrated the use of CNNs to transfer the artistic style of one image to another, as shown in the following examples:

Figure 5.1 – (A) Content image. (B)-(D) Bottom image is the style image and the bigger pictures are stylized images (Source: Gatys et al., 2015, “A Neural Algorithm of Artistic Style” https://arxiv.org/abs/1508.06576)

Unlike most deep learning trainings that require tons of training data, neural style transfer requires only two images – content and style images. We can use pre-trained CNN such as VGG to transfer the style from the style image to the content image.

As shown...

Improving style transfer

The research community and industry were excited about neural style transfer and wasted no time in putting it to use. Some set up websites to allow users to upload photos to perform style transfer, while some used that to create merchandise to sell. Then people realized some of the shortcomings of the original neural style transfer and worked to improve it.

One of the biggest limitations is that style transfer takes all the style information, including the color and brush strokes of the entire style image, and transfers it to the whole of the content image. Using the examples that we just did in the previous section, the blueish color from the style image was transferred into both the building and background. Wouldn't it be nice if we had the choice to transfer only the brush stroke but not the color, and just to the preferred regions?

The lead author of neural style transfer and his team produced a new algorithm to address these issues. The following...

Arbitrary style transfer in real time

In this section, we will learn how to implement a network that could perform arbitrary style transfer in real time. We have already learned how to use a feed-forward network for faster inference and that solves the real-time part. We have also learned how to use conditional instance normalization to transfer a fixed number of styles. Now, we will learn one further normalization technique that allows for any arbitrary style, and then we are good to go in terms of implementing the code.

Implementing adaptive instance normalization

Like CIN, AdaIN is also instance normalization, meaning that the mean and standard deviation are calculated across (H, W) per image, and per channel, as opposed to batch normalization, which calculates across (N, H, W). In CIN, the gammas and betas are trainable variables, and they learn the means and variances that are needed for different styles. In AdaIN, gammas and betas are replaced by standard deviations and...

Introduction to style-based GANs

The innovations in style transfer made their way into influencing the development of GANs. Although GANs at that time could generate realistic images, they were generated by using random latent variables, where we had little understanding in terms of what they represented. Even though multimodal GANs could create variations in generated images, we did not know how to control the latent variables to achieve the outcome that we wanted.

In an ideal world, we would love to have some knobs to independently control the features we would like to generate, as in the face manipulation exercise in Chapter 2, Variational Autoencoder. This is known as disentangled representation, which is a relatively new idea in deep learning. The idea of disentangled representation is to separate an image into independent representation. For example, a face has two eyes, a nose, and a mouth, with each of them being a representation of a face. As we have learned in style transfer...

Summary

In this chapter, we covered the evolution of styled-based generative models. It all started with neural style transfer, where we learned that the image can be disentangled into content and style. The original algorithm was slowed and the iterative optimization process in inference time replaced with a feed-forward style transfer that could perform style transfer in real time.

We then learned that the Gram matrix is not the only method for representing style, and that we could use the layers' statistics instead. As a result, normalization layers have been explored to control the style of an image, which eventually led to the creation of AdaIN. By combing a feed-forward network and AdaIN, we implemented arbitrary style transfer in real time.

With the success in style transfer, AdaIN found its way into GANs. We went over the MUNIT architecture in detail in terms of how AdaIN was used for multimodal image generation. There is a style-based GAN that you should be familiar...