Chapter 7: Neural Style Transfer

In the previous chapter, we started exploring generative models using PyTorch. We built machine learning models that can generate text and music by training the models without supervision on text and music data, respectively. We will continue exploring generative modeling in this chapter by applying a similar methodology to image data.

We will mix different aspects of two different images, A and B, to generate a resultant image, C, that contains the content of image A and the style of image B. This task is also popularly known as neural style transfer because, in a way, we are transferring the style of image B to image A in order to achieve image C, as illustrated in the following figure:

Figure 7.1 – Neural style transfer example

First, we will briefly discuss how to approach this problem and understand the idea behind achieving style transfer. Using PyTorch, we will then implement our own neural style transfer...

Technical requirements

We will be using Jupyter notebooks for all of our exercises.

Here's the list of Python libraries that must be installed for this chapter using pip. For example, here, you must run pip install torch==1.4.0 on the command line and so on:

jupyter==1.0.0
torch==1.4.0
torchvision==0.5.0
matplotlib==3.1.2
Pillow==8.0.1

All the code files that are relevant to this chapter are available at https://github.com/PacktPublishing/Mastering-PyTorch/tree/master/Chapter07.

Understanding how to transfer style between images

In Chapter 3, Deep CNN Architectures, we discussed convolutional neural networks (CNNs) in detail. CNNs are largely the most successful class of models when working with image data. We have seen how CNN-based architectures are the best-performing architectures of neural networks on tasks such as image classification, object detection, and so on. One of the core reasons behind this success is the ability of convolutional layers to learn spatial representations.

For example, in a dog versus cat classifier, the CNN model is essentially able to capture the content of an image in its higher-level features, which helps it detect dog-specific features against cat-specific features. We will leverage this ability of an image classifier CNN to grasp the content of an image.

We know that VGG is a powerful image classification model, as discussed in Chapter 3, Deep CNN Architectures. We are going to use the convolutional part of the VGG...

Implementing neural style transfer using PyTorch

Having discussed the internals of a neural style transfer system, we are all set to build one using PyTorch. In the form of an exercise, we will load a style and a content image. Then, we will load the pre-trained VGG model. After defining which layers to compute the style and content loss on, we will trim the model so that it only retains the relevant layers. Finally, we will train the neural style transfer model in order to refine the generated image epoch by epoch.

Loading the content and style images

In this exercise, we will only show the important parts of the code for demonstration purposes. To access the full code, go to https://github.com/PacktPublishing/Mastering-PyTorch/blob/master/Chapter07/neural_style_transfer.ipynb. Follow these steps:

1. Firstly, we need to import the necessary libraries by running the following lines of code:

   from PIL import Image
   import matplotlib.pyplot as pltimport torch
   import torch.nn...

Summary

In this chapter, we applied the concept of generative machine learning to images by generating an image that contains the content of one image and the style of another – a task known as neural style transfer. First, we understood the idea behind the style transfer algorithm, especially the use of the gram matrix in order to extract styles from an image.

Next, we used PyTorch to build our own neural style transfer model. We used parts of a pre-trained VGG19 model to extract content and style information through some of its convolutional layers. We replaced the max pooling layers of the VGG19 model with average pooling layers for a smooth gradient flow. We then input a random initial image to the style transfer model and with the help of a style and a content loss, we fine-tuned the image pixels using gradient descent.

This input image evolves over epochs and gives us the final generated image, which contains the content of the content image and style of the style...