CycleGAN - Turn Paintings into Photos

CycleGAN is a type of Generative Adversarial Network (GAN) for cross-domain transfer tasks, such as changing the style of an image, turning paintings into photos, and vice versa, photo enhancement, changing the season of a photo, and many more. CycleGANs were introduced by Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros in a paper entitled: Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. This was produced in February 2018 at the Berkeley AI Research (BAIR) laboratory, UC Berkeley, which is available at the following link: https://arxiv.org/pdf/1703.10593.pdf. CycleGANs caused a stir in the GAN community because of their widespread use cases. In this chapter, we will be working with CycleGANs and, specifically, using them to turn paintings into photos.

In this chapter, we will cover the following...

An introduction to CycleGANs

To turn photos into paintings or paintings, into photos, normal GANs require a pair of images. A CycleGAN is a type of GAN network that can translate an image from one domain X, to another domain Y, without the need for paired images. A CycleGAN tries to learn a generator network, which, in turn, learns two mappings. Instead of training a single generator network used in most of the GANs, CycleGANs train two generator and two discriminator networks.

There are two generator networks in a CycleGAN, which are as follows:

1. Generator A: Learns a mapping , where X is the source domain and Y is the target domain. It takes an image from the source domain A, and converts it into an image that is similar to an image from the target domain B. Basically, the aim of the network is to learn a mapping so that G(X) is similar to Y.
2. Generator B: Learns a mapping ...

Setting up the project

If you haven't already cloned the repository with the complete code for all chapters, clone the repository now. The downloaded code has a directory called Chapter07, which contains the entire code for this chapter. Execute the following commands to set up the project:

1. Start by navigating to the parent directory as follows:

cd Generative-Adversarial-Networks-Projects

2. Now, change the directory from the current directory to Chapter07, as shown in the following example:

cd Chapter07

3. Next, create a Python virtual environment for this project, as shown in the following code:

virtualenv venv
virtualenv venv -p python3 # Create a virtual environment using 
           python3 interpreter
virtualenv venv -p python2 # Create a virtual environment using 
           python2 interpreter

We will be using this newly created virtual environment for this project....

Downloading the dataset

In this chapter, we will be working with the monet2photo dataset. This dataset is open source and is made available by the Berkeley AI Research (BAIR) laboratory, UC Berkeley. You can choose to download the dataset manually from the following link: https://people.eecs.berkeley.edu/~taesung_park/CycleGAN/datasets/monet2photo.zip.

After downloading it, unzip it in the root directory.

Alternatively, to automatically download the dataset, execute the following commands:

wget https://people.eecs.berkeley.edu/~taesung_park/CycleGAN/datasets/monet2photo.zip
upzip monet2photo.zip

These commands will download the dataset and unzip it in the project's root directory.

The monet2photo dataset is available for educational purposes only. To use it in commercial projects, you have to get permission from the BAIR laboratory, UC Berkeley. We don't hold the copyright...

Keras implementation of CycleGAN

As discussed earlier in this chapter in the An Introduction to CycleGANs section, CycleGANs have two network architectures, a generator and a discriminator network. In this section, we will write the implementation for all the networks.

Before starting to write the implementations, however, create a Python file, main.py, and import the essential modules, as follows:

from glob import glob
import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf
from keras import Input, Model
from keras.layers import Conv2D, BatchNormalization, Activation, Add, Conv2DTranspose, \
    ZeroPadding2D, LeakyReLU
from keras.optimizers import Adam
from keras_contrib.layers import InstanceNormalization
from scipy.misc import imread, imresize

The generator network

...

Training the CycleGAN

We have already covered the training objective function in the An Introduction to CycleGANs section. We have also created the respective Keras models for both networks. Training the CycleGAN is a multi-step process. We will perform the following steps to train the network:

1. Loading the dataset
2. Creating the generator and the discriminator networks
3. Training the network for a specified number of epochs
4. Plotting the losses
5. Generating new images

Let's define the essential variables before starting to train the network, as follows:

data_dir = "/Path/to/dataset/directory/*.*"
batch_size = 1
epochs = 500

Loading the dataset

Before doing anything else, load the dataset by performing the following...

Practical applications of CycleGANs

There are many applications of CycleGANs. In this chapter, we have used a CycleGAN to turn paintings into photos. They can also be used in the following cases:

• Style transfer: For example, turning photos into paintings and vice versa, turning pictures of horses into zebras and vice versa, and turning pictures of oranges into pictures of apples and vice versa
• Photo enhancement: CycleGANs can be used to enhance the quality of pictures
• Season transfer: For example, turning a picture of winter into a picture of summer and vice versa
• Game style transfer: CycleGANs can be used to transfer the style of Game A to Game B

Summary

In this chapter, we have learned how to turn paintings into photos using a CycleGAN. We started with an introduction to CyleGANs and explored the architectures of networks involved in CycleGANs. We also explored the different loss functions required to train CycleGANs. This was followed by an implementation of CycleGAN in the Keras framework. We trained the CycleGAN on the monet2photo dataset and visualized the generated images, the losses, and the graphs for different networks. Before concluding the chapter, we explored the real-world use cases of CycleGANs.

In the next chapter, we will work on the pix2pix network for image-to-image translation. In pix2pix, we will explore conditional GANs for image translation.

Further reading

CycleGAN has many known use cases. Try to explore new uses using the following articles for assistance:

• Turning Fortnite into PUBG with Deep Learning (CycleGAN): https://towardsdatascience.com/turning-fortnite-into-pubg-with-deep-learning-cyclegan-2f9d339dcdb0
• GAN — CycleGAN (Playing magic with pictures): https://medium.com/@jonathan_hui/gan-cyclegan-6a50e7600d7
• Introduction to CycleGANs: https://medium.com/coding-blocks/introduction-to-cyclegans-1dbdb8fbe781
• Understanding and Implementing CycleGAN in TensorFlow: https://hardikbansal.github.io/CycleGANBlog/