Conditional GAN - Image-to-Image Translation Using Conditional Adversarial Networks

Pix2pix is a type of Generative Adversarial Network (GAN) that is used for image-to-image translation. Image-to-image translation is a method for translating one representation of an image into another representation. Pix2pix learns a mapping from input images into output images. It can be used to convert black and white images to color images, sketches to photographs, day images to night images, and satellite images to map images. The pix2pix network was first introduced in the paper titled Image-to-Image Translation with Conditional Adversarial Networks, by Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros; which can be found at the following link: https://arxiv.org/pdf/1611.07004.pdf.

In this chapter, we will cover the following topics:

• Introducing the Pix2pix network
• The architecture...

Introducing Pix2pix

Pix2pix is a variant of the conditional GAN. We have already covered conditional GANs in Chapter 3, Face-Aging Using Conditional GAN (cGAN). Before moving forward, make sure you take a look at what cGANs are. Once you are comfortable with cGANs, you can continue with this chapter. Pix2pix is a type of GAN that is capable of performing image-to-image translation using the unsupervised method of machine learning (ML). Once trained, pix2pix can translate an image from domain A to domain B. Vanilla CNNs can also be used for image-to-image translation, but they don't generate realistic and sharp images. On the other hand, pix2pix has shown immense potential to be able to generate realistic and sharp images. We will be training pix2pix to translate labels of facades to images of facade. Let's start by understanding the architecture of pix2pix.

...

Setting up the project

If you haven't already cloned the repository with the complete code for all chapters, clone the repository now. The cloned repository has a directory called Chapter09, which contains the entire code for this chapter. Execute the following commands to set up the project:

1. Start by navigating to the parent directory as follows:

cd Generative-Adversarial-Networks-Projects

2. Now change the directory from the current directory to Chapter09:

cd Chapter09

3. Next, create a Python virtual environment for this project:

virtualenv venv
virtualenv venv -p python3 # Create a virtual environment using python3 interpreter
virtualenv venv -p python2 # Create a virtual environment using python2 interpreter

We will be using this newly created virtual environment for this project. Each chapter has its own separate virtual environment.

4. Next, activate the newly created...

Preparing the data

In this chapter, we will be working with the Facades dataset, which is available at the following link:

http://efrosgans.eecs.berkeley.edu/pix2pix/datasets/facades.tar.gz.

This dataset contains facade labels and ground truth facade images. A facade is generally the front side of a building, and facade labels are architectural labels of a facade image. We will learn more about facades after we download the dataset. Perform the following commands to download and extract the dataset:

1. Download the dataset by executing the following commands:

# Before downloading the dataset navigate to data directory
cd data

# Download the dataset
wget http://efrosgans.eecs.berkeley.edu/pix2pix/datasets/facades.tar.gz

2. After downloading the dataset, extract the dataset using the following command:

tar -xvzf facades.tar.gz

The file structure of the downloaded dataset is as follows...

A Keras implementation of pix2pix

As mentioned, pix2pix has two networks: a generator and a discriminator. The generator is inspired by the architecture of U-Net. Similarly, the discriminator network is inspired by the architecture of PatchGAN. We will implement both networks in the following sections.

Before starting to write the implementations, create a Python file main.py and import the essential modules as follows:

import os
import time

import h5py
import keras.backend as K
import matplotlib.pyplot as plt
import numpy as np
from cv2 import imwrite
from keras import Input, Model
from keras.layers import Convolution2D, LeakyReLU, BatchNormalization, UpSampling2D, Dropout,     Activation, Flatten, Dense, Lambda, Reshape, concatenate
from keras.optimizers import Adam

The generator network

...

Training the pix2pix network

Like any other GAN, training the pix2pix network is a two-step process. In the first step, we train the discriminator network. In the second step, we train the adversarial network, which eventually trains the generator network. Let's start training the network.

Perform the following steps to train an SRGAN network:

1. Start by defining the hyperparameters that are required for training:

epochs = 500
num_images_per_epoch = 400
batch_size = 1
img_width = 256
img_height = 256
num_channels = 1
input_img_dim = (256, 256, 1)
patch_dim = (256, 256)

# Specify dataset directory path
dataset_dir = "pix2pix-keras/pix2pix/data/facades_bw"

2. Next, define the common optimizer, shown as follows:

common_optimizer = Adam(lr=1E-4, beta_1=0.9, beta_2=0.999,  
                        epsilon=1e-08)

For all networks, we will use the Adam optimizer with the learning...

Practical applications of a pix2pix network

There are many applications of a pix2pix network. These include the following:

• To convert pixel level segmentation into real images
• To convert day images into night images and vice versa
• To convert satellite areal images into map images
• To convert sketches into photos
• To convert black and white images into colored images and vice versa

The following is an image that was taken from the official paper indicated. It shows the different use cases of a pix2pix network:

Source: Image-to-Image Translation with Conditional Adversarial Networks:

Source: arXiv:1611.07004 [cs.CV]

Summary

In this chapter, we have learned what a pix2pix network is and explored its architecture. We started by downloading and preparing the dataset for training, and then prepared the project and looked at a Keras implementation of a pix2pix network. After that, we looked at the objective function for the training of a pix2pix network. We then trained the pix2pix network on the facades dataset and explored some practical applications of a pix2pix network.

In the next chapter, we will be predicting the future of GANs. We are going to be looking at what might happen in the GAN domain in the near future and how it might change our industries and our daily lives.