Chapter 9
Bayesian Additive Regression Trees

Individually, we are one drop. Together, we are an ocean. – Ryunosuke Satoro

In the last chapter, we discussed the Gaussian process (GPs), a non-parametric model for regression. In this chapter, we will learn about another non-parametric model for regression known as Bayesian additive regression trees, or BART to friends. We can consider BART from many different perspectives. It can be an ensemble of decision trees, each with a distinct role and contribution to the overall understanding of the data. These trees, guided by Bayesian priors, work harmoniously to capture the nuances of the data, avoiding the pitfall of individual overfitting. Usually, BART is discussed as a standalone model, and software that implements it is usually limited to one or a few models. In this chapter, we will take a different approach and use PyMC-BART, a Python library that allows the use of BART models within PyMC.

In this chapter, we will cover the...

9.1 Decision trees

Before jumping into BART models, let’s take a moment to discuss what decision trees are. A decision tree is like a flowchart that guides you through different questions until you reach a final choice. For instance, suppose you need to decide what type of shoes to wear every morning. To do so, you may ask yourself a series of questions. ”Is it warm?” If yes, you then ask something more specific, like ”Do I have to go to the office?” Eventually, you will stop asking questions and reach an output value like flip-flops, sneakers, boots, moccasins, etc.

This flowchart can be conveniently encoded in a tree structure, where at the root of the tree we place more general questions, then proceed along the tree to more and more specific ones, and finally arrive at the leaves of the tree with the output of the different types of shoes. Trees are very common data structures in computer science and data analysis.

More formally, we can say that...

9.2 BART models

A Bayesian additive regression trees (BART) model is a sum of m trees that we use to approximate a function [Chipman et al., 2010]. To complete the model, we need to set priors over trees. The main function of such priors is to prevent overfitting while retaining the flexibility that trees provide. Priors are designed to keep the individual trees relatively shallow and the values at the leaf nodes relatively small.

PyMC does not support BART models directly but we can use PyMC-BART, a Python module that extends PyMC functionality to support BART models. PyMC-BART offers:

• A BART random variable that works very similar to other distributions in PyMC like pm.Normal, pm.Poisson, etc.

• A sampler called PGBART as trees cannot be sampled with PyMC’s default step methods such as NUTS or Metropolis.

• The following utility functions to help work with the result of a BART model:

  • pmb.plot_pdp: A function to generate partial dependence plots [Friedman, ...

9.3 Distributional BART models

As we saw in Chapter 6, for generalized linear models, we are not restricted to creating linear models for the mean or location parameter; we can also model other parameters, for example, the standard deviation of a Gaussian or even both the mean and standard deviation. The same applies to BART models.

To exemplify this, let’s model the bike dataset. We will use rented as the response variable and hour, temperature, humidity, and workday as predictor variables. As we did previously, we are going to use a NegativeBinomial distribution as likelihood. This distribution has two parameters μ and alpha. We are going to use a sum of trees for both parameters. The following code block shows the model:

Code 9.5

with pm.Model() as model_bb: 
    μ = pmb.BART("μ", X, np.log(Y), shape=(2, 348), separate_trees=True) 
    pm.NegativeBinomial('yl', np.exp(μ...

9.4 Constant and linear response

By default, PyMC-BART will fit trees that return a single value at each leaf node. This is a simple approach that usually works just fine. However, it is important to understand its implications. For instance, this means that predictions for any value outside the range of the observed data used to fit the model will be constants. To see this, go back and check Figure 9.2. This tree will return 1.9 for any value below c1. Notice that this will still be the case if we, instead, sum a bunch of trees, because summing a bunch of constant values results in yet another constant value.

Whether this is a problem or not is up to you and the context in which you apply the BART model. Nevertheless, PyMC-BART offers a response argument that you pass to the BART random variable. Its default value is "constant". You can change it to "linear", in which case PyMC-BART will return a linear fit at each leaf node or "mix", which will propose...

9.5 Choosing the number of trees

The number of trees (m) controls the flexibility of the BART function. As a rule of thumb, the default value of 50 should be enough to get a good approximation. And larger values, like 100 or 200, should provide a more refined answer. Usually, it is hard to overfit by increasing the number of trees, because the larger the number of trees, the smaller the values at the leaf nodes.

In practice, you may be worried about overshooting m because the computational cost of BART, both in terms of time and memory, will increase. One way to tune m is to perform K-fold cross-validation, as recommended by Chipman et al. [2010]. Another option is to approximate cross-validation by using LOO as discussed in Chapter 5. We have observed that LOO can indeed be of help to provide a reasonable value of m [Quiroga et al., 2022].

9.6 Summary

BART is a flexible non-parametric model where a sum of trees is used to approximate an unknown function from the data. Priors are used to regularize inference, mainly by restricting trees’ learning capacity so that no individual tree is able to explain the data, but rather the sum of trees. PyMC-BART is a Python library that extends PyMC to work with BART models.

We built a few BART models in this chapter, and learned how to perform variable selection and use partial dependence plots and individual conditional plots to interpret the output of BART models.

9.7 Exercises

1. Explain each of the following:

   • How is BART different from linear regression and splines?

   • When might you want to use linear regression over BART?

   • When might you want to use Gaussian processes over BART?

2. In your own words, explain why it can be the case that multiple small trees can fit patterns better than one single large tree. What is the difference in the two approaches? What are the trade-offs?

3. Below, we provide two simple synthetic datasets. Fit a BART model with m=50 to each of them. Plot the data and the mean fitted function. Describe the fit.

   • x = np.linspace(-1, 1., 200) and y = np.random.normal(2*x, 0.25)

   • x = np.linspace(-1, 1., 200) and y = np.random.normal(x**2, 0.25)

   • Create your own synthetic dataset.

4. Create the following dataset Y = 10sin(πX₀X₁)+20(X₂ −0.5)² +10X₃ +5X₄ + , where ∼(0,1) and X_0:9 ∼(0,1). This is called Friedman’s five-dimensional function. Notice that we actually have 10 dimensions, but the last 5...

Join our community Discord space

Join our Discord community to meet like-minded people and learn alongside more than 5000 members at: https://packt.link/bayesian