Definition and purpose

First, we can consider a common definition you may find online:

Boosting is a machine learning ensemble meta-algorithm for primarily reducing bias, and also variance in supervised learning, and a family of machine learning algorithms which convert weak learners to strong ones.                                                                                                                    -Wikipedia                         https://en.wikipedia.org/wiki/Boosting_(machine_learning)

Note

Reminder: In statistics, ensemble methods use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the fundamental or basic learning algorithms (although results vary by data and data model).

Before we head into the details behind statistical boosting, it is imperative that we take some time here to first understand bias, variance, noise, and what is meant by a weak learner, and a strong learner.

The following sections will cover these terms...