Chapter 7. Using Simulated Images To Create Photo-Realistic Eyeballs with SimGAN

In this chapter, we'll cover the following recipes:

• How the SimGAN architecture works
• Pseudocode – how does it work?
• How to work with training data
• Code implementation – loss functions
• Code implementation – generator
• Code implementation – discriminator
• Code implementation – GAN
• Training the SimGAN network

Introduction

This chapter will focus on the SimGAN paper and how to take simulated data and make it look more realistic. The generator network used in the SimGAN architecture is able to improve the fidelity of simulated data.

 

How SimGAN architecture works

Apple previously released a paper titled Learning from Simulated and Unsupervised Images through Adversarial Training (https://arxiv.org/pdf/1612.07828.pdf), in which authors coined the architecture type SimGAN. As set out in the paper, SimGAN allows users to refine simulated data to make it look more realistic. In this section, we'll discuss how SimGAN architecture works.

Getting ready

The only thing you'll need in this section is the paper previously mentioned, which can be downloaded and read at: https://arxiv.org/pdf/1612.07828.pdf titled Learning from Simulated and Unsupervised Images through Adversarial Training.

How to do it...

In the SimGAN paper, authors set out to create a refiner network that can accurately improve the realism of synthetic images in an unsupervised manner. In the past, it has been quite hard to find matched simulation and real data for training such networks, but SimGAN has changed the existing landscape thanks to its focus on a simulated...

Pseudocode – how does it work?

With every technique, we need to understand the baseline algorithm before we can lay down any code. So, in this section, we'll discuss how the training algorithm works.

Getting ready

In this section, we'll be referring to the SimGAN paper once again.

How to do it...

In the SimGAN paper, the authors provided a convenient graphic for users to base their development on. We already know that we need to develop models for each of the networks, but how do we train a network in the first place? The following diagram offers an explanation:

Algorithm 

Let's convert the preceding diagram into the following, tangible steps:

1. Read both synthetic images and real images into variables.
2. Then, for every epoch, do the following:
   • Train the refiner networks on a random mini batch for K_Gtimes
   • Train the discriminator network on a random mini batch for K_D times
3. Stop when the number of epochs reached, or lost, has not changed significantly for nepochs.

 

How to work with training data

As with every architecture we train throughout this book, understanding the structure of the data and the development environment is important to overall success. So, in this section, we'll set up the development environment and download the data inside the Docker container.

Getting ready

You'll need to create a folder at the $HOME directory level of your Linux machine with the following directory structure (which can be checked using the tree function):

├── docker
│   ├── build.sh
│   ├── clean.sh
│   ├── Dockerfile
│   └── kaggle.json
├── out
├── README.md
├── run.sh
└── src

How to do it...

In this chapter, we're going to introduce the Kaggle API so we can grab the necessary data for the SimGAN training architecture. Using the Kaggle API will require you to set up a Kaggle account and get API token access.

Kaggle and its API

Kaggle.com is a popular online site that holds machine learning (ML) competitions and discussions. Kaggle also supplies an API for accessing...

Code implementation – loss functions

In this section, we're going to develop custom loss functions that will be used for the discriminator, generator, and adversarial models. We'll cover two loss functions in this section, which we'll go over in detail.

Getting ready

It's time for a directory check! Make sure you've created and placed the relevant data in each of the following folders and files. In this step, we're adding the loss.py file:

├── data
├── docker
│   ├── build.sh
│   ├── clean.sh
│   ├── Dockerfile
│   └── kaggle.json
├── out
├── README.md
├── run.sh
└── src
    ├── loss.py

How to do it...

This is a fairly simple section made up of three primary steps—creating the loss.py file and placing two loss functions in it for us to inherit later on in the development.

Perform the following steps to create the loss.py file:

1. Add the python3 interpreter to the top of the file and import tensorflow, as follows:

#!/usr/bin/env python3
import tensorflow as tf

2. Implement the self-regularization loss...

Code implementation – generator

In this case, the generator in also known as the refiner network. This generator, therefore, is the networkthat will take and refine the simulated data.

Getting ready

Check that you have the following files in the correct place:

├── data
├── docker
│   ├── build.sh
│   ├── clean.sh
│   ├── Dockerfile
│   └── kaggle.json
├── out
├── README.md
├── run.sh
└── src
    ├── generator.py
    loss.py

How to do it...

In this section, we'll look at build boilerplate items, model development, and helper functions in order to help us to build the full generator.

Boilerplate items

There are two key steps in the boilerplate, and they are as follows:

1. Add all of the following import statements needed to create the generator (refiner) network:

#!/usr/bin/env python
import sys
import numpy as np
from keras.layers import Dense, Reshape, Input, BatchNormalization, Concatenate, Activation
from keras.layers.core import Activation
from keras.layers.convolutional import UpSampling2D, Convolution2D...

Code implementation – discriminator

The discriminator in SimGAN is a fairly simple Convolutional Neural Network (CNN) with a small twist at the end—it outputs the likelihood of simulated and real. In this section, we'll also make use of a function from the loss class we built earlier.

Getting ready

We've built a set of loss functions and the generator class, so now it's time to build the discriminator class. You should see the following structure in your directory:

├── data
├── docker
│   ├── build.sh
│   ├── clean.sh
│   ├── Dockerfile
│   └── kaggle.json
├── imgs
│   ├── create_token.png
│   ├── kaggle_signup.png
│   ├── MyAccount.png
│   ├── refiner_network_training.png
│   └── simGAN_network.png
├── out
│   └── Generator_Model.png
├── README.md
├── run.sh
└── src
 ├── discriminator.py
 ├── generator.py
 ├── loss.py

How to do it...

The discriminator is very similar to other discriminators we've built in previous chapters. In this case, we're essentially building a CNN with a slightly different...

Code implementation – GAN

The Generative Adversarial Model, or GAN, is at the heart of adversarial training architecture. In fact, this model is different only in the fact that we use custom loss functions in our compile step. Let's take a look at how it's implemented.

 

 

Getting ready

This section will fill out the core of the base classes and functionality we need to have for training the simGAN architecture. The following files, and structure, should be included in your current directory:

├── data
├── docker
│   ├── build.sh
│   ├── clean.sh
│   ├── Dockerfile
│   └── kaggle.json
├── out
├── README.md
├── run.sh
└── src
    ├── discriminator.py
    ├── gan.py
    ├── generator.py
    ├── loss.py

How to do it...

The GAN model is vastly simplified in comparison to the building of the generator and discriminator. Essentially, this class will put the generator and discriminator into adversarial training along with the custom loss functions.

Take the following steps:

1. Use the python3 interpreter and...

Training the simGAN network

Now that we've built the infrastructure, we can develop the training methodology in the train script. In this section, we'll also create the run python and shell scripts that will be used for running everything in the Docker environment.

Getting ready

We're almost at the end! So, make sure you have every one of the following directories and files in your $HOME directory:

├── data
├── docker
│   ├── build.sh
│   ├── clean.sh
│   ├── Dockerfile
│   └── kaggle.json
├── out
│   ├── GAN_Model.png
│   └── Generator_Model.png
├── README.md
├── run.sh
└── src
    ├── discriminator.py
    ├── gan.py
    ├── generator.py
    ├── loss.py
    ├── run.py
    └── train.py

How to do it...

The training script will read in data, process the data for input into the networks, and then train the simGAN model.

Initialization

Take the following steps to initialize the training class and the basic functionality needed to train the models:

 

 

1. Create a train.py file and place the following imports...

Exercise

1. Create a way to pre-train the refiner and discriminator networks.