Chapter 9
Next Steps in Bayesian Deep Learning

Throughout this book, we’ve covered the fundamental concepts behind Bayesian deep learning (BDL), from understanding what uncertainty is and its role in developing robust machine learning systems, right through to learning how to implement and analyze the performance of several fundamental BDL. While what you’ve learned will equip you to start developing your own BDL solutions, the field is moving quickly, and there are many new techniques on the horizon.

To wrap up the book, in this chapter we’ll take a look at the current trends in BDL, before we dive into some of the latest developments in the field. We’ll conclude by introducing some alternatives to BDL, and provide some advice on additional resources you can use to continue your journey into Bayesian machine learning methods.

We’ll cover the following sections:

• Current trends in BDL

• How are BDL methods being applied to solve real-world problems...

9.1 Current trends in BDL

In this section, we’ll explore the current trends in BDL. We’ll look at which models are particularly popular in the literature and discuss why certain models have been selected for certain applications. This should give you a good imdivssion of how the fundamentals covered throughout the book apply more broadly across a variety of application domains.

Figure 9.1: Popularity of key BDL search terms over time

As we see in Figure 9.1, there’s been a marked increase in popularity of search terms related to BDL over the past decade. Unsurprisingly, this follows the trend in the popularity of deep learning search terms, as we see in Figure 9.2; as deep learning has become more popular, there has been increased interest in quantifying the uncertainty associated with the predictions produced by DNNs. Interestingly, these plots both show a similar dip in popularity in mid-late 2021, indicating that as long as deep learning is popular...

9.2 How are BDL methods being applied to solve real-world problems?

Just as deep learning is having an impact on a diverse variety of application domains, BDL is becoming an increasingly important tool, particularly where large amounts of data are being used within safety-critical or mission-critical systems. In these cases – as is the case for most real-world applications – being able to quantify when models ”know they don’t know” is crucial to developing reliable and robust systems.

One significant application area for BDL is in safety-critical systems. In their 2019 paper titled Safe Reinforcement Learning with Model Uncertainty Estimates, Björn Lütjens et al. demonstrate that the use of BDL methods can produce safer behavior in collision-avoidance scenarios (the inspiration for our reinforcement learning example in Chapter 8, Applying Bayesian Deep Learning).

Similarly, in the paper Uncertainty-Aware Deep Learning for Safe Landing...

9.3 Latest methods in BDL

In this book, we’ve introduced some of the core techniques used within BDL: Bayes by Backprop (BBB), Probabilistic Backpropagation (PBP), Monte-Carlo dropout (MC dropout), and deep ensembles. Many BNN approaches you’ll encounter in the literature will be based on one of these methods, and having these under your belt provides you with a versatile toolbox of approaches for developing your own BDL solutions. However, as with all aspects of machine learning, the field of BDL is progressing rapidly, and new techniques are being developed on a regular basis. In this section, we’ll explore a selection of recent developments from the field.

9.3.1 Combining MC dropout and deep ensembles

Why use just one Bayesian neural network technique when you could use two? This is exactly the approach taken by University of Edinburgh researchers Remus Pop and Patric Fulop in their paper, Deep Ensemble Bayesian Active Learning: Addressing the Mode Collapse...

9.4 Alternatives to Bayesian deep learning

While the focus of the book is on Bayesian inference with DNNs, these aren’t always the best choice for the job. Generally speaking, they’re a great choice when you have large amounts of high dimensional data. As we discussed in Chapter 3, Fundamentals of Deep Learning (and as you probably know), deep networks excel in these scenarios, and thus adapting them for Bayesian inference is a sensible choice. On the other hand, if you have small amounts of low-dimensional data (with tens of features, fewer than 10,000 data points), then you may be better off with more traditional, well-principled Bayesian inference, such as via sampling or GPs.

That said, there has been interest in scaling GPs, and the research community has developed GP-based methods that both scale to large amounts of data and are capable of complex non-linear transformations. In this section, we’ll introduce these alternatives in case you wish to pursue...

9.5 Your next steps in BDL

Throughout this chapter, we’ve concluded our introduction to BDL by taking a look at a variety of techniques that could help you to improve on the fundamental methods explored in the book. We’ve also taken a look at how the powerful gold-standard of Bayesian inference – the GP – can be adapted to tasks generally reserved for deep learning. While it is indeed possible to adapt GPs to these tasks, we also advise that it’s generally easier and more practical to use the methods presented in this book, or methods derived from them. As always, it’s up to you as the machine learning engineer to determine what is best for the task at hand, and we are confident that the material from the book will equip you well for the challenges ahead.

While this book provides you with the necessary fundamentals to get started, there’s always more learn – particularly in such a rapidly moving field! In the next section, we’...

9.6 Further reading

The following reading recommendations are provided for those who wish to learn more about the recent methods presented in this chapter. These give a great insight into current challenges in the field, looking beyond Bayesian neural networks and into scalable Bayesian inference more generally:

• Deep Ensemble Bayesian Active Learning, Pop and Fulop: This paper demonstrates the advantages of combining deep ensembles with MC dropout to produce better-calibrated uncertainty estimates, as shown when applying their method to active learning tasks.

• Uncertainty in Neural Networks: Approximately Bayesian Ensembling, Pearce et al.: This paper introduces a simple and effective method for improving the performance of deep ensembles. The authors show that by promoting diversity through a simple adaptation to the loss function, the ensemble is able to produces better-calibrated uncertainty estimates.

• Sparse Gaussian Processes Using Pseudo-Inputs, Snelson and Gharamani: This paper...