Preface

In recent years, Deep Learning has made unprecedented success stories in difficult problems in vision, speech, natural language processing and understanding, and all other areas with abundance of data. The interest in this field from companies, universities, governments, and research organizations has accelerated the advances in the field. This book covers select important topics in Deep Learning with three new chapters, Object Detection, Semantic Segmentation, and Unsupervised Learning using Mutual Information. The advanced theories are explained by giving a background of the principles, digging into the intuition behind the concepts, implementing the equations and algorithms using Keras, and examining the results.

Artificial Intelligence (AI), as it stands today, is still far from being a well-understood field. Deep Learning (DL), as a sub field of AI, is in the same position. While it is far from being a mature field, many real-world applications such as vision-based detection and recognition, autonomous navigation, product recommendation, speech recognition and synthesis, energy conservation, drug discovery, finance, and marketing are already using DL algorithms. Many more applications will be discovered and built. The aim of this book is to explain advanced concepts, give sample implementations, and let the readers as experts in their field identify the target applications.

A field that is not completely mature is a double-edged sword. On one edge, it offers a lot of opportunities for discovery and exploitation. There are many unsolved problems in deep learning. This translates into opportunities to be the first to market – be that in product development, publication, or recognition. The other edge is it would be difficult to trust a not-fully-understood field in a mission-critical environment. We can safely say that if asked, very few machine learning engineers will ride an auto-pilot plane controlled by a deep learning system. There is a lot of work to be done to gain this level of trust. The advanced concepts that are discussed in this book have a high chance of playing a major role as the foundation in gaining this level of trust.

No DL book will be able to completely cover the whole field. This book is not an exception. Given time and space, we could have touched interesting areas like natural language processing and understanding, speech synthesis, automated machine learning (AutoML), graph neural networks (GNNs), Bayesian deep learning, and many others. However, this book believes in choosing and explaining select areas so that readers can take up other fields that are not covered.

As the reader who is about to embark upon reading this book, keep in mind that you chose an area that is exciting and can have a huge impact on society. We are fortunate to have a job that we look forward to working on as we wake up in the morning.

Who this book is for

The book is intended for machine learning engineers and students who would like to gain a better understanding of advanced topics in deep learning. Each discussion is supplemented with code implementation in Keras. In particular, the Keras API of TensorFlow 2 or simply tf.keras is what's used This book is for readers who would like to understand how to translate theory into working code implementation in Keras. Apart from understanding theories, code implementation is usually one of the difficult tasks in applying machine learning to real-world problems.

What this book covers

Chapter 1, Introducing Advanced Deep Learning with Keras, covers the key concepts of deep learning such as optimization, regularization, loss functions, fundamental layers, and networks and their implementation in tf.keras. This chapter serves as a review of both deep learning and tf.keras using the sequential API.

Chapter 2, Deep Neural Networks, discusses the functional API of tf.keras. Two widely used deep network architectures, ResNet and DenseNet, are examined and implemented in tf.keras using the functional API.

Chapter 3, Autoencoders, covers a common network structure called the autoencoder, which is used to discover the latent representation of input data. Two example applications of autoencoders, denoising and colorization, are discussed and implemented in tf.keras.

Chapter 4, Generative Adversarial Networks (GANs), discusses one of the recent significant advances in deep learning. GAN is used to generate new synthetic data that appear real. This chapter explains the principles of GAN. Two examples of GAN, DCGAN and CGAN, are examined and implemented in tf.keras.

Chapter 5, Improved GANs, covers algorithms that improve the basic GAN. The algorithms address the difficulty in training GANs and improve the perceptual quality of synthetic data. WGAN, LSGAN, and ACGAN are discussed and implemented in tf.keras.

Chapter 6, Disentangled Representation GANs, discusses how to control the attributes of the synthetic data generated by GANs. The attributes can be controlled if the latent representations are disentangled. Two techniques in disentangling representations, InfoGAN and StackedGAN, are covered and implemented in tf.keras.

Chapter 7, Cross-Domain GANs, covers a practical application of GAN, translating images from one domain to another, commonly known as cross-domain transfer. CycleGAN, a widely used cross-domain GAN, is discussed and implemented in tf.keras. This chapter demonstrates CycleGAN performing colorization and style transfer.

Chapter 8, Variational Autoencoders (VAEs), discusses another important topic in DL. Similar to GAN, VAE is a generative model that is used to produce synthetic data. Unlike GAN, VAE focuses on decodable continuous latent space that is suitable for variational inference. VAE and its variations, CVAE and β-VAE, are covered and implemented in tf.keras.

Chapter 9, Deep Reinforcement Learning, explains the principles of reinforcement learning and Q-learning. Two techniques in implementing Q-learning for discrete action space are presented, Q-table update and Deep Q-Networks (DQNs). Implementation of Q-learning using Python and DQN in tf.keras are demonstrated in OpenAI Gym environments.

Chapter 10, Policy Gradient Methods, explains how to use neural networks to learn the policy for decision making in reinforcement learning. Four methods are covered and implemented in tf.keras and OpenAI Gym environments, REINFORCE, REINFORCE with Baseline, Actor-Critic, and Advantage Actor-Critic. The example presented in this chapter demonstrates policy gradient methods on a continuous action space.

Chapter 11, Object Detection, discusses one of the most common applications of computer vision, object detection or identifying and localizing objects in an image. Key concepts of a multi-scale object detection algorithm called SSD are covered and an implementation is built step by step using tf.keras. An example technique for dataset collection and labeling is presented. Afterward, the tf.keras implementation of SSD is trained and evaluated using the dataset.

Chapter 12, Semantic Segmentation, discusses another common application of computer vision, semantic segmentation or identifying the object class of each pixel in an image. Principles of segmentation are discussed. Then, semantic segmentation is covered in more detail. An example implementation of a semantic segmentation algorithm called FCN is built and evaluated using tf.keras. The same dataset collected in the previous chapter is used but relabeled for semantic segmentation.

Chapter 13, Unsupervised Learning Using Mutual Information, looks at how DL is not going to advance if it heavily depends on human labels. Unsupervised learning focuses on algorithms that do not require human labels. One effective technique to achieve unsupervised learning is to take advantage of the concept of Mutual Information (MI). By maximizing MI, unsupervised clustering/classification is implemented and evaluated using tf.keras.

To get the most out of this book

• Deep learning and Python: The reader should have a fundamental knowledge of deep learning and its implementation in Python. While previous experience in using Keras to implement deep learning algorithms is important, it is not required. Chapter 1, Introducing Advanced Deep Learning with Keras, offers a review of deep learning concepts and their implementation in tf.keras.
• Math: The discussions in this book assume that the reader is familiar with calculus, linear algebra, statistics, and probability at college level.
• GPU: The majority of the tf.keras implementations in this book require a GPU. Without a GPU, it is not practical to execute many of the code examples because of the time involved (many hours to days). The examples in this book use reasonable amounts of data as much as possible in order to minimize the use of high-performance computers. The reader is expected to have access to at least NVIDIA GTX 1060.
• Editor: The example code in this book was edited using vim in Ubuntu Linux 18.04 LTS and MacOS Catalina. Any Python-aware text editor is acceptable.
• TensorFlow 2: The code examples in this book are written using the Keras API of TensorFlow 2 or tf2. Please ensure that the NVIDIA GPU driver and tf2 are both properly installed.
• GitHub: We learn by example and experimentation. Please git pull or fork the code bundle for the book from its GitHub repository. After getting the code, examine it. Run it. Change it. Run it again. Do creative experiments by tweaking the code. It is the only way to appreciate all the theory explained in the chapters. Giving a star on the book's GitHub repository https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras is also highly appreciated.

Download the example code files

The code bundle for the book is hosted on GitHub at:

https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Download the color images

We also provide color images of figures used in this book. You can download it here: https://static.packt-cdn.com/downloads/9781838821654_ColorImages.pdf.

Conventions used

The code in this book is in Python. More specifically, Python 3. For example:

A block of code is set as follows:

def build_generator(inputs, image_size):
    """Build a Generator Model
    Stack of BN-ReLU-Conv2DTranpose to generate fake images
    Output activation is sigmoid instead of tanh in [1].
    Sigmoid converges easily.
    Arguments:
        inputs (Layer): Input layer of the generator 
            the z-vector)
        image_size (tensor): Target size of one side
            (assuming square image)
    Returns:
        generator (Model): Generator Model
    """
    image_resize = image_size // 4
    # network parameters 
    kernel_size = 5
    layer_filters = [128, 64, 32, 1]
    x = Dense(image_resize * image_resize * layer_filters[0])(inputs)
    x = Reshape((image_resize, image_resize, layer_filters[0]))(x)
    for filters in layer_filters:
        # first two convolution layers use strides = 2
        # the last two use strides = 1
        if filters > layer_filters[-2]:
            strides = 2
        else:
            strides = 1
        x = BatchNormalization()(x)
        x = Activation('relu')(x)
        x = Conv2DTranspose(filters=filters,
                            kernel_size=kernel_size,
                            strides=strides,
                            padding='same')(x)
    x = Activation('sigmoid')(x)
    generator = Model(inputs, x, name='generator')
    return generator

When we wish to draw your attention to a particular part of a code block,the relevant lines or items are set in bold:

# generate fake images
fake_images = generator.predict([noise, fake_labels])
# real + fake images = 1 batch of train data
x = np.concatenate((real_images, fake_images))
# real + fake labels = 1 batch of train data labels
labels = np.concatenate((real_labels, fake_labels))

Whenever possible, docstrings are is included. At the very least, text comments are used to minimize space usage.

Any command-line code execution is written as follows:

python3 dcgan-mnist-4.2.1.py

The above example has the following layout: algorithm-dataset-chapter.section.number.py. The command-line example is DCGAN on the MNIST dataset in Chapter 4, Generative Adversarial Networks (GANs) second section and first listing. In some cases, the explicit command line to execute is not written but it is assumed to be:

python3 name-of-the-file-in-listing

The file name of the code example is included in the Listing caption. This book uses Listing to identify code examples in the text.

Bold: Indicates a new term, an important word, or words that you see on the screen, for example, in menus or dialog boxes, also appear in the text like this. For example: StackedGAN has two additional loss functions, Conditional and Entropy.

Warnings or important notes appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: Email [e-mail], and mention the book's title in the subject of your message. If you have questions about any aspect of this book, please email us at [e-mail].

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book we would be grateful if you would report this to us. Please visit, http://www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.

Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [e-mail] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit http://authors.packtpub.com.

Reviews

Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you!

For more information about Packt, please visit packtpub.com.