Classification and Regression with Trees

Tree based algorithms are very popular for two reasons: they are interpretable, and they make sound predictions that have won many machine learning competitions on online platforms, such as Kaggle. Furthermore, they have many use cases outside of machine learning for solving problems, both simple and complex.

Building a tree is an approach to decision-making used in almost all industries. Trees can be used to solve both classification- and regression-based problems, and have several use cases that make them the go-to solution!

This chapter is broadly divided into the following two sections:

• Classification trees
• Regression trees

Each section will cover the fundamental theory of different types of tree based algorithms, along with their implementation in scikit-learn. By the end of this chapter, you will have learned how to aggregate several...

Technical requirements

You will be required to have Python 3.6 or greater, Pandas ≥ 0.23.4, Scikit-learn ≥ 0.20.0, and Matplotlib ≥ 3.0.0 installed on your system.

The code files of this chapter can be found on GitHub:
https://github.com/PacktPublishing/Machine-Learning-with-scikit-learn-Quick-Start-Guide/blob/master/Chapter_06.ipynb.

Check out the following video to see the code in action:

http://bit.ly/2SrPP7R

Classification trees

Classification trees are used to predict a category or class. This is similar to the classification algorithms that you have learned about previously in this book, such as the k-nearest neighbors algorithm or logistic regression.

Broadly speaking, there are three tree based algorithms that are used to solve classification problems:

• The decision tree classifier
• The random forest classifier
• The AdaBoost classifier

In this section, you will learn how each of these tree based algorithms works, in order to classify a row of data as a particular class or category.

The decision tree classifier

The decision tree is the simplest tree based algorithm, and serves as the foundation for the other two algorithms....

Regression trees

You have learned how trees are used in order to classify a prediction as belonging to a particular class or category. However, trees can also be used to solve problems related to predicting numeric outcomes. In this section, you will learn about the three types of tree based algorithms that you can implement in scikit-learn in order to predict numeric outcomes, instead of classes:

• The decision tree regressor
• The random forest regressor
• The gradient boosted tree

The decision tree regressor

When we have data that is non-linear in nature, a linear regression model might not be the best model to choose. In such situations, it makes sense to choose a model that can fully capture the non-linearity of such data...

Ensemble classifier

The concept of ensemble learning was explored in this chapter, when we learned about random forests, AdaBoost, and gradient boosted trees. However, this concept can be extended to classifiers outside of trees.

If we had built a logistic regression, random forest, and k-nearest neighbors classifiers, and we wanted to group them all together and extract the final prediction through majority voting, then we could do this by using the ensemble classifier.

This concept can be better understood with the aid of the following diagram:

Ensemble learning with a voting classifier to predict fraud transactions

When examining the preceding diagram, note the following:

• The random forest classifier predicted that a particular transaction was fraudulent, while the other two classifiers predicted that the transaction was not fraudulent.
• The voting classifier sees that two...

Summary

While this chapter was rather long, you have entered the world of tree based algorithms, and left with a wide arsenal of tools that you can implement in order to solve both small- and large-scale problems. To summarize, you have learned the following:

• How to use decision trees for classification and regression
• How to use random forests for classification and regression
• How to use AdaBoost for classification
• How to use gradient boosted trees for regression
• How the voting classifier can be used to build a single model out of different models

In the upcoming chapter, you will learn how you can work with data that does not have a target variable or labels, and how to perform unsupervised machine learning in order to solve such problems!