Chapter 7
Practical Considerations for Bayesian Deep Learning

Over the last two chapters, Chapter 5, Principled Approaches for Bayesian Deep Learning and Chapter 6, Using the Standard Toolbox for Bayesian Deep Learning, we’ve been introduced to a range of methods that facilitate Bayesian inference with neural networks. Chapter 5, Principled Approaches for Bayesian Deep Learning introduced specially crafted Bayesian neural network approximations, while Chapter 6, Using the Standard Toolbox for Bayesian Deep Learning showed how we can use the standard toolbox of machine learning to add uncertainty estimates to our models. These families of methods come with their own advantages and disadvantages. In this chapter, we will explore some of these differences in practical scenarios in order to help you understand how to select the best method for the task at hand.

We will also look at different sources of uncertainty, which can improve your understanding of the...

7.1 Technical requirements

To complete the practical tasks in this chapter, you will need a Python 3.8 environment with the SciPy and scikit-learn stack and the following additional Python packages installed:

• TensorFlow 2.0

• TensorFlow Probability

All of the code for this book can be found on the GitHub repository for the book: https://github.com/PacktPublishing/Enhancing-Deep-Learning-with-Bayesian-Inference.

7.2 Balancing uncertainty quality and computational considerations

While Bayesian methods have many benefits, there are also trade-offs to consider in terms of memory and computational overheads. These considerations play a critical role in selecting the most appropriate methods to use within real-world applications.

In this section, we’ll examine the trade-offs between different methods in terms of performance and uncertainty quality, and we’ll learn how we can use TensorFlow’s profiling tools to measure the computational costs associated with different models.

7.2.1 Setting up our experiments

To evaluate the performance of different models, we’ll need a few different datasets. One of these is the California Housing dataset, which is conveniently provided by scikit-learn. The others we’ll use are commonly used in papers comparing uncertainty models: the Wine Quality dataset and the Concrete Comdivssive Strength dataset. Let’s take a look...

7.3 BDL and sources of uncertainty

In this case study, we will look at how we can model aleatoric and epistemic uncertainty in a regression problem when we are trying to predict a continuous outcome variable. We will use a real-life dataset of diamonds that contains the physical attributes of more than 50,000 diamonds as well as their prices. In particular, we will look at the relationship between the weight of a diamond (measured as its carat) and the price paid for the diamond.

Step 1: Setting up the environment

To set up the environment, we import several packages. We import tensorflow and tensorflow_probability for building and training vanilla and probabilistic neural networks, tensorflow_datasets for importing the diamonds data set, numpy for performing calculations and operations on numerical arrays (such as calculating the mean), pandas for handling DataFrames, and matplotlib for plotting:

 
import matplotlib.pyplot as plt  
import numpy as np ...

7.4 Summary

In this chapter, we’ve taken a look at a number of practical considerations of using Bayesian deep learning: exploring trade-offs in model performance and learning how we can use Bayesian neural network methods to better understand the effects of different uncertainty sources on our data.

In the next chapter, we’ll dig further into applying BDL through a variety of case studies, demonstrating the benefits of these methods in a range of practical settings.

7.5 Further reading

• Practical Considerations for Probabilistic Backpropagation, Matt Benatan et al.: In this paper, the authors explore methods to get the most out of PBP, demonstrating how different early stopping approaches can be used to improve training, exploring the tradeoffs associated with mini-batching, and more

• Modeling aleatoric and epistemic uncertainty using TensorFlow and TensorFlow Probability, Alexander Molak: In this Jupyter notebook, the author shows how to model aleatoric and epistemic uncertainty on regression toy data

• Weight Uncertainty in Neural Networks, Charles Blundell et al.: In this paper, the authors introduce BBB, which we use in the regression case study and is one of the key pieces of BDL literature

• Deep Deterministic Uncertainty: A Simple Baseline, Jishnu Mukhoti et al.: In this work, the authors describe several experiments related to the different types of uncertainty and introduce the AmbiguousMNIST dataset that we used in the last case study

• Uncertainty...