Chapter 3. Introducing Bayesian Inference

In Chapter 1, Introducing the Probability Theory, we learned about the Bayes theorem as the relation between conditional probabilities of two random variables such as A and B. This theorem is the basis for updating beliefs or model parameter values in Bayesian inference, given the observations. In this chapter, a more formal treatment of Bayesian inference will be given. To begin with, let us try to understand how uncertainties in a real-world problem are treated in Bayesian approach.

Bayesian view of uncertainty

The classical or frequentist statistics typically take the view that any physical process-generating data containing noise can be modeled by a stochastic model with fixed values of parameters. The parameter values are learned from the observed data through procedures such as maximum likelihood estimate. The essential idea is to search in the parameter space to find the parameter values that maximize the probability of observing the data seen so far. Neither the uncertainty in the estimation of model parameters from data, nor the uncertainty in the model itself that explains the phenomena under study, is dealt with in a formal way. The Bayesian approach, on the other hand, treats all sources of uncertainty using probabilities. Therefore, neither the model to explain an observed dataset nor its parameters are fixed, but they are treated as uncertain variables. Bayesian inference provides a framework to learn the entire distribution of model parameters, not just...

Exercises

1. Derive the equation for the posterior mean by expanding the square in the exponential for each i, collecting all similar power terms, and making a perfect square again. Note that the product of exponentials can be written as the exponential of a sum of terms.

2. For this exercise, we use the dataset corresponding to Smartphone-Based Recognition of Human Activities and Postural Transitions, from the UCI Machine Learning repository (https://archive.ics.uci.edu/ml/datasets/Smartphone-Based+Recognition+of+Human+Activities+and+Postural+Transitions). It contains values of acceleration taken from an accelerometer on a smartphone. The original dataset contains x, y, and z components of the acceleration and the corresponding timestamp values. For this exercise, we have used only the two horizontal components of the acceleration x and y. In this exercise, let's assume that the acceleration follows a normal distribution. Let's also assume a normal prior distribution for the mean values of...

References

1. Berger J.O. Statistical Decision Theory and Bayesian Analysis. Springer Series in Statistics. 1993. ISBN-10: 0387960988

2. Jayes E.T. Probability Theory: The Logic of Science. Cambridge University Press. 2003. ISBN-10: 052159271

3. Wittman D. "Fisher Matrix for Beginners". Physics Department, University of California at Davis (http://www.physics.ucdavis.edu/~dwittman/Fisher-matrix-guide.pdf)

4. Metropolis N, Rosenbluth A.W., Rosenbluth M.N., Teller A.H., Teller E. "Equations of State Calculations by Fast Computing Machines". Journal of Chemical Physics 21 (6): 1087–1092. 1953

5. Geman S., Geman D. "Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images". IEEE Transactions on Pattern Analysis and Machine Intelligence 6 (6): 721-741. 1984

Summary

In this chapter, we covered basic principles of Bayesian inference. Starting with how uncertainty is treated differently in Bayesian statistics compared to classical statistics, we discussed deeply various components of Bayes' rule. Firstly, we learned the different types of prior distributions and how to choose the right one for your problem. Then we learned the estimation of posterior distribution using techniques such as MAP estimation, Laplace approximation, and MCMC simulations. Once the readers have comprehended this chapter, they will be in a position to apply Bayesian principles in their data analytics problems. Before we start discussing specific Bayesian machine learning problems, in the next chapter, we will review machine learning in general.