Chapter 4
Introducing Bayesian Deep Learning

In Chapter 2, Fundamentals of Bayesian Inference, Fundamentals of Bayesian Inference, we saw how traditional methods for Bayesian inference can be used to produce model uncertainty estimates, and we introduced the properties of well-calibrated and well-principled methods for uncertainty estimation. While these traditional methods are powerful in many applications, Chapter 2, Fundamentals of Bayesian Inference also highlighted some of their limitations with respect to scaling. In Chapter 3, Fundamentals of Deep Learning, we saw the impressive things DNNs are capable of given large amounts of data; but we also learned that they aren’t perfect. In particular, they often lack robustness for out-of-distribution data – a major concern when we consider the deployment of these methods in real-world applications.

Figure 4.1: BDL combines the strengths of both deep learning and traditional Bayesian inference

BDL...

4.1 Technical requirements

To complete the practical tasks in this chapter, you will need a Python 3.8 environment with the SciPy stack and the following additional Python packages installed:

• TensorFlow 2.0

• TensorFlow Probability

• Seaborn plotting library

All of the code for this book can be found in the GitHub repository for the book: https://github.com/PacktPublishing/Enhancing-Deep-Learning-with-Bayesian-Inference.

4.2 The ideal BNN

As we saw in the previous chapter, a standard neural network comprises multiple layers. Each of these layers comprises a number of perceptrons – and these perceptrons comprise a multiplicative component (weight) and an additive component (bias). Each weight and bias parameter comprises a single parameter – or point estimate – and, in combination, these parameters transform the input to the perceptron. As we’ve seen, multiple layers of perceptrons are capable of achieving impressive feats when trained via backpropagation. However, these point estimates contain very limited information – let’s take a look.

Generally speaking, the goal of deep learning is to find (potentially very, very many) parameter values that best map a set of inputs onto a set of outputs. That is, given some data, for each parameter in our network, we’ll choose the parameter that best describes the data. This often boils down to taking the mean –...

4.3 BDL fundamentals

Throughout the rest of the book, we will introduce a range of methods necessary to make BDL possible. There are a number of common themes present through these methods. We’ll cover these here, so that we have a good understanding of these concepts when we encounter them later on.

These concepts include the following:

• Gaussian assumptions: With many BDL methods, we use Gaussian assumptions to make things computationally tractable

• Uncertainty sources: We’ll take a look at the different sources of uncertainty, and how we can determine the contributions of these sources for some BDL methods

• Likelihoods: We were introduced to likelihoods in Chapter 2, Fundamentals of Bayesian Inference, and here we’ll learn more about the importance of likelihood as a metric for evaluating the calibration of probabilistic models

Let’s look at each of these in the following subsections.

4.3.1 Gaussian assumptions

In the ideal case described previously...

4.4 Tools for BDL

In this chapter, as well as in Chapter 2, Fundamentals of Bayesian Inference, we’ve seen a lot of equations involving probability. While it’s possible to create BDL models without a probability library, having a library that supports some of the fundamental functions makes things much easier. As we’re using TensorFlow for the examples in this book, we’ll be using the TensorFlow Probability (TFP) library to help us with some of these probabilistic components. In this section, we’ll introduce TFP and show how it can be used to easily implement many of the concepts we’ve seen in Chapter 2, Fundamentals of Bayesian Inference and Chapter 4, Introducing Bayesian Deep Learning.

Much of the content up to this point has been about introducing the concept of working with distributions. As such, the first TFP module we’ll learn about is the distributions module. Let’s take a look:

 
import tensorflow_probability...

4.5 Summary

In this chapter, we were introduced to the fundamental concepts that we’ll need to progress through the book and learn how to implement and use BNNs. Most crucially, we learned about the ideal BNN, which introduced us to the core ideas underlying BDL, and the computational difficulties of achieving this in practice. We also covered the fundamental practical methods used in BDL, giving us a grounding in the concepts that allow us to implement computationally tractable BNNs.

The chapter also introduced the concept of uncertainty sources, describing the difference between data and model uncertainty, how these contribute to total uncertainty, and how we can estimate the contributions of different types of uncertainty with various models. We also introduced one of the most fundamental components in probabilistic inference – the likelihood function – and learned about how it can help us to train better principled and better calibrated models. Lastly, we were...

4.6 Further reading

This chapter has introduced the material necessary to start working with BDL; however, there are many resources that go into more depth on the topics of uncertainty sources. The following are a few recommendations for readers interested in exploring the theory and code in more depth:

• Machine Learning: A Probabilistic Perspective, Murphy: Kevin Murphy’s extremely popular book on machine learning has become a staple for students and researchers in the field. This book provides a detailed treatment of machine learning from a probabilistic standpoint, unifying concepts from statistics, machine learning, and Bayesian probability.

• TensorFlow Probability Tutorials: in this book, we’ll see how TensorFlow Probability can be used to develop BNNs, but their website includes a wide array of tutorials addressing probabilistic programming more generally: https://www.tensorflow.org/probability/overview

• Pyro Tutorials: Pyro is a PyTorch-based library for probabilistic...