Style Transfer with GANs

Neural networks are improving in a number of tasks involving analytical and linguistic skills. Creativity is one sphere where humans have had an upper hand. Not only is art subjective and has no defined boundaries, it is also difficult to quantify. Yet this has not stopped researchers from exploring the creative capabilities of algorithms. There have been several successful attempts at creating, understanding, and even copying art or artistic styles over the years, a few examples being Deep Dream¹ and Neural Style Transfer.²

Generative models are well suited to tasks associated with imagining and creating. Generative Adversarial Networks (GANs) in particular have been studied and explored in detail for the task of style transfer over the years. One such example is presented in Figure 7.1, where the CycleGAN architecture has been used to successfully transform photographs into paintings using the styles of famous artists such as Monet and Van Gogh.

...

Paired style transfer using pix2pix GAN

In Chapter 6, Image Generation with GANs, we discussed a number of innovations related to GAN architectures that led to improved results and better control of the output class. One of those innovations was conditional GANs. This simple yet powerful addition to the GAN setup enabled us to navigate the latent vector space and control the generator to generate specific outputs. We experimented with a simple MNIST conditional GAN where we were able to generate the output of our choice.

In this section, we will cover a variant of conditional GANs in the context of style transfer. We will go through details of the pix2pix architecture, discuss the important components and also train a paired style transfer network of our own. We will close this section with some amazing and innovative use cases of such a capability.

Style transfer is an intriguing research area, pushing the boundaries of creativity and deep learning together. In their work...

Unpaired style transfer using CycleGAN

Paired style transfer is a powerful setup with a number of use cases, some of which we discussed in the previous section. It provides the ability to perform cross-domain transfer given a pair of source and target domain datasets. The pix2pix setup also showcased the power of GANs to understand and learn the required loss functions without the need for manually specifying them.

While being a huge improvement over hand-crafted loss functions and previous works, paired style transfer is limited by the availability of paired datasets. Paired style transfer requires the input and output images to be structurally the same even though the domains are different (aerial to map, labels to scene, and so on). In this section, we will focus on an improved style transfer architecture called CycleGAN.

CycleGAN improves upon paired style transfer architecture by relaxing the constraints on input and output images. CycleGAN explores the unpaired style...

Related works

Style transfer is an amusing field and a lot of parallel research is going on across different research groups to improve the state of the art. The two most influential works in the paired and unpaired style transfer space have been discussed in this chapter so far. There have been a few more related works in this space that are worth discussing.

In this section, we will briefly discuss two more works in the unpaired image-to-image translation space that have similar ideas to CycleGAN. Specifically, we will touch upon the DiscoGAN and DualGAN setups, as they present similar ideas with minor changes.

It is important to note that there are a number of other works in the same space. We limit our discussion to only a few of them for the sake of completeness and consistency. Readers are encouraged to explore other interesting architectures as well.

DiscoGAN

Kim and Cha et al. presented a model that discovers cross-domain relations with GANs called DiscoGAN...

Summary

In this chapter, we explored the creative side of GAN research through the lenses of image-to-image translation tasks. While the creative implications are obvious, such techniques also open up avenues to improve the research and development of computer vision models for domains where datasets are hard to get.

We started off the chapter by understanding the paired image-to-image translation task. This task provides training data where the source and destination domains have paired training samples. We explored this task using the pix2pix GAN architecture. Through this architecture, we explored how the encoder-decoder architecture is useful for developing generators that can produce high-fidelity outputs. The pix2pix paper took the encoder-decoder architecture one step further by making use of skip-connections or a U-Net style generator.

This setup also presented another powerful concept, called the Patch-GAN discriminator, which works elegantly to assist the overall...

References

1. Mordvintsev, A., McDonald, K., Rudolph, L., Kim, J-S., Li, J., & daviga404. (2015). deepdream. GitHub repository. https://github.com/google/deepdream
2. Gatys, L.A., Ecker, A.S., & Bethge, M. (2015). A Neural Algorithm of Artistic Style. arXiv. https://arxiv.org/abs/1508.06576
3. Zhu, J-Y., Park, T., Isola, P., & Efros, A.A. (2017). Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. arXiv. https://arxiv.org/abs/1703.10593
4. Isola, P., Zhu, J-Y., Zhou, T., & Efros, A.A. (2017). Image-to-Image Translation with Conditional Adversarial Networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 5967-5976. https://ieeexplore.ieee.org/document/8100115
5. Ronneberger, O., Fisher, P., & Brox, T. (2015). U-net: Convolutional Networks for Biomedical Image Segmentation. MICCAI, 2015. https://arxiv.org/abs/1505.04597
6. Kim, T., Cha, M., Kim, H., Lee, J.K., & Kim, J. (2017)...