Chapter 2. Classification – Decision Tree Learning

In the previous chapter, we discussed different types of machine learning, including supervised classification tasks; in this chapter, we will build our first Swift application for this. We will discuss main components of machine learning development stack, and will also exercise in data generation, exploratory analysis, preprocessing, and models training and evaluation in Python. After this, we will transfer our model to Swift. We will also discuss a specific class of supervised learning algorithms—decision tree learning and its extension: random forest.

The following topics are waiting for us in this chapter:

• Machine learning software development stack
• Python toolbox for machine learning: IPython, SciPy, scikit-learn
• Dataset generation and exploratory analysis
• Data preprocessing
• Decision tree learning and random forest
• Assessing the model performance using different performance metrics
• Underfitting and overfitting
• Exporting scikit-learn models...

Machine learning toolbox

For many years, the programming language of choice for machine learning was one of the following: Python, R, MATLAB, C++. This is not due to some specific language features, but because of the infrastructure around it: libraries and tools. Swift is a relatively young programming language, and anyone who chooses it as a primary tool for machine learning development should start from the very basic building blocks, and build his own tools and libraries. Recently, Apple became more open to third-party Python machine learning tools: Core ML can work with some of them.

Here is a list of components that are needed for the successful machine learning research and development, and examples of popular libraries and tools of the type:

• Linear algebra: Machine learning developer needs data structures like vectors, matrices, and tensors with compact syntax and hardware-accelerated operations on them. Examples in other languages: NumPy, MATLAB, and R standard libraries, Torch.
• Probability...

Prototyping the first machine learning app

Usually, before implementing a machine learning application for mobile devices, you want to do a quick and dirty prototype just to check your ideas. This allows to save a lot of time when you realize that the model you initially thought works perfectly for your problem, in reality doesn't. The quickest way to do a prototype is to use Python or R tools listed in the previous section.

Python is a general-purpose programming language with rich infrastructure and vibrant community. Its syntax is similar in many ways to Swift's one. Throughout this book, we'll use it for prototyping, and Swift for actual development.

When you have tested your ideas and a model prototype works as you expect, you can start thinking about how to port it to an iOS. You have several options here:

Inference-only options:

• Check the Core ML, and a list of the Python libraries it supports. Maybe, you will be able to export your model in Core ML format, and run it on a device.
• Write...

IPython notebook crash course

Feel free to skip this section if you're familiar with the Python and Jupyter notebooks.

IPython notebook and its web-based GUI Jupyter are standard tools for data-driven machine learning development. Jupyter is also a handy tool for learning Python and its libraries. You can combine pieces of code with comments in markdown format. You can also execute pieces of code in place, chaining them one after another, and immediately seeing the results of computations. It also allows to embed interactive charts, tables, videos, and other multimedia objects inside the notebook. We will use Jupyter notebooks for writing quick prototypes of our models.

To create a new notebook, run in the Terminal:

> jupyter notebook

You will see output similar to this:

[I 10:51:23.269 NotebookApp] Serving notebooks from local directory: ...[I 10:51:23.269 NotebookApp] 0 active kernels [I 10:51:23.270 NotebookApp] The Jupyter Notebook is running at: http://localhost:8888/?token=3c073db5636e366fd750e661cc597652025fdbf41162c125...

Time to practice

In the following sections, we'll dive into machine learning practice, to get a feeling of what it looks like. Just like in a theater play, in machine learning you have a list of characters and a list of acts.

Two main characters are:

• Dataset
• Model

Three main acts are:

• Dataset preparation
• Model training
• Model evaluation

We'll go through all these acts, and by the end of the chapter we'll have our first trained model. First, we need to define a problem, and then we can start coding a prototype in Python. Our destination point is a working model in Swift. Don't take the problem itself too seriously, though, because as the first exercise, we're going to solve a fictional problem.

Machine learning for extra-terrestrial life explorers

Swift is undoubtedly the programming language of the future. In the nearest years, we're expecting to see Swift being employed to program-intelligent scout robots that will explore alien planets and life forms on them. These robots should be able to recognize and classify aliens they will encounter. Let's build a model to distinguish between two alien species using their characteristic features.

The biosphere of the distant planet consists mainly of two species: night predators rabbosauruses, and peaceful, herbivorous platyhogs (see the following diagram). Roboscouts are equipped with sensors to measure only three features of each individual: length (in meters), color, and fluffiness.

Figure 2.1: Objects of interest in our first machine learning task. Picture by Mykola Sosnovshchenko.

Note

The full code of the Python part of this chapter can be found here: ML_Intro.ipynb.

Loading the dataset

Create and open a new IPython notebook. In the chapter's supplementary materials, you can see the file extraterrestrials.csv. Copy it to the same folder where you created your notebook. In the first cell of your notebook, execute the magical command:

In []: 
%matplotlib inline 

This is needed to see inline plots right in the notebook in the future.

The library we are using for datasets loading and manipulation is pandas. Let's import it, and load the .csv file:

In []: 
import pandas as pd 
df = pd.read_csv('extraterrestrials.csv', sep='t', encoding='utf-8', index_col=0) 

Object df is a data frame. This is a table-like data structured for efficient manipulations over the different data types. To see what's inside, execute:

In []: 
df.head() 
Out[]: 

	Length	Color	Fluffy	Label
0	27.545139	Pink gold	True	Rabbosaurus
1	12.147357	Pink gold	False	Platyhog
2	23.454173	Light black	True	Rabbosaurus
3	29.956698	Pink gold	True	Rabbosaurus
4	34.884065	Light black	True	Rabbosaurus

This prints the first five rows of the...

Exploratory data analysis

First, we want to see how many individuals of each class we have. This is important, because if the class distribution is very imbalanced (like 1 to 100, for example), we will have problems training our classification models. You can get data frame columns via the dot notation. For example, df.label will return you the label column as a new data frame. The data frame class has all kinds of useful methods for calculating the summary statistics. The value_counts() method returns the counts of each element type in the data frame:

In []: 
df.label.value_counts() 
Out[]: 
platyhog       520 
rabbosaurus    480 
Name: label, dtype: int64 

The class distribution looks okay for our purposes. Now let's explore the features.

We need to group our data by classes, and calculate feature statistics separately to see the difference between the creature classes. This can be done using the groupby() method. It takes the label of the column by which you want to group your data:

In [...

Data preprocessing

In the following sections we will take a look at the different data processing techniques.

Converting categorical variables

As you already have noticed, a data frame can contain columns with the data of different types. To see which type has each column, we can check the dtypes attribute of the data frame. You can think about Python attributes as being similar to Swift properties:

In []: 
df.dtypes 
Out[]: 
length    float64 
color      object 
fluffy       bool 
label      object 
dtype: object 

While length and fluffy columns contain the expected datatypes, the types of color and label are less transparent. What are those objects? This means those columns can contain any type of the object. At the moment, we have strings in them, but what we really want them to be are categorical variables. In case you don't remember from the previous chapter, categorical variables are like Swift enums. Fortunately for us, data frame has handy methods for converting columns from one type...

Decision trees everywhere

The algorithm that we're going to use for our first machine learning exercise is called a decision tree classifier. A decision tree is a set of rules that describe the process of decision making (see figure 2.5 for example).

Decision trees are widely used outside the machine learning in different domains; for example, in business analysis. The popularity of decision trees is understandable: they are easy to interpret, and nice to visualize. For many years, they were built manually using the domain expert knowledge. Fortunately, now we have machine learning algorithms that can easily turn almost any labeled dataset into a decision tree.

Training the decision tree classifier

Let's learn how to train the decision tree classifier as shown in the following code snippet:

In []: 
from sklearn import tree 
tree_model = tree.DecisionTreeClassifier(criterion='entropy', random_state=42) 
tree_model = tree_model.fit(X_train, y_train) 
tree_model 
Out[]: 
DecisionTreeClassifier(class_weight=None,  
            criterion='entropy', max_depth=None, 
            max_features=None, max_leaf_nodes=None, 
            min_impurity_split=1e-07, min_samples_leaf=1, 
            min_samples_split=2, min_weight_fraction_leaf=0.0, 
            presort=False, random_state=42, splitter='best') 

The most interesting for us are the class attributes of DecisionTreeClassifier:

• criterion: The way to estimate the best partition (see the How decision tree learning works section).
• max_depth: Maximum tree depth.
• max_features: The maximum number of attributes to account in one split.
• min_samples_leaf: The minimum number of objects in the leaf; for example, if it...

How decision tree learning works

Decision tree learning is a supervised, non-parametric algorithm used for classification and regression.

Building a tree automatically from data

The Twenty Questions game is a traditional game where one of the players is the answerer who chooses an object (or a famous person in some variants), not revealing what it is to the other participants. All the other players are trying to guess what the object is by asking questions like Can I eat this? or Is it a human? where answers can only be yes or no.

If you have never heard about this game, refer to Wikipedia: https://en.wikipedia.org/wiki/Twenty_Questions.

This is essentially a tree learning algorithm. To win in a game, you should pose such questions that discriminate the most; for example, the question, Is it alive? in the beginning of the game is clearly better than Is it a cucumber?. This ability to dissect the hypothesis space in an optimal way is formalized in the notion of information gain criterion.

Combinatorial...

Implementing first machine learning app in Swift

You can transfer your model from Python to Swift in two ways: transfer a trained model, or train a model from the ground up in Swift. The first option is easy in the case of decision trees, as a trained model can be expressed as a set of if-else conditions, which is trivial to code manually. Training the model from the ground up is required only in the situation where you want your app to learn in runtime. We will stick to the first approach in this example, but instead of coding rules manually, we will export the scikit-learn model for iOS using Core ML tools.

Introducing Core ML

Core ML was first presented at Apple WWDC 2017. Defining Core ML as machine learning framework is not fair, because it lacks learning capabilities; it's rather a set of conversion scripts to plug the pre-trained model into your Apple applications. Still, it is an easy way for newcomers to start running their first models on iOS.

Core ML features

Here is a list of Core ML features:

• coremltools Python package includes several converters for popular machine learning frameworks: scikit-learn, Keras, Caffe, LIBSVM, and XGBoost.
• Core ML framework allows running inference (making predictions) on a device. Scikit-learn converter also supports some data transformation and model pipelining.
• Hardware acceleration (Accelerate framework and Metal under the hood).
• Supports iOS, macOS, tvOS, and watchOS.
• Automatic code generation for OOP-style interoperability with Swift.

The biggest Core ML limitation is that it doesn't support models training.

Exporting the model for iOS

In our Jupyter notebook...

Summary

In this chapter, we had our first experience of building a machine learning application, starting from the data and all the way over to the working iOS application. We went through several phases in this chapter:

• Exploratory data analysis using Jupyter, pandas, and Matplotlib
• Data preparation—splitting, and handling categorical variables
• Model prototyping using scikit-learn
• Model tuning and evaluation
• Porting prototype for the mobile platform using Core ML
• Model validation on a mobile device

There are several machine learning topics that we've learned about in this chapter: model parameters vs. hyperparameters, overfitting vs. underfitting, evaluation metrics: cross-validation, accuracy, precision, recall, and F1-score. These are the basic things that will be recurring topics throughout this book.

We've become acquainted with two machine learning algorithms, namely decision trees and random forest, a type of model ensemble.

In the next chapter, we're going to continue exploring classification...

The rest of the chapter is locked

Exploratory data analysis

First, we want to see how many individuals of each class we have. This is important, because if the class distribution is very imbalanced (like 1 to 100, for example), we will have problems training our classification models. You can get data frame columns via the dot notation. For example, df.label will return you the label column as a new data frame. The data frame class has all kinds of useful methods for calculating the summary statistics. The value_counts() method returns the counts of each element type in the data frame:

In []: 
df.label.value_counts() 
Out[]: 
platyhog       520 
rabbosaurus    480 
Name: label, dtype: int64 

The class distribution looks okay for our purposes. Now let's explore the features.

We need to group our data by classes, and calculate feature statistics separately to see the difference between the creature classes. This can be done using the groupby() method. It takes the label of the column by which you want to group your data:

In [...

Data preprocessing

In the following sections we will take a look at the different data processing techniques.

Converting categorical variables

As you already have noticed, a data frame can contain columns with the data of different types. To see which type has each column, we can check the dtypes attribute of the data frame. You can think about Python attributes as being similar to Swift properties:

In []: 
df.dtypes 
Out[]: 
length    float64 
color      object 
fluffy       bool 
label      object 
dtype: object 

While length and fluffy columns contain the expected datatypes, the types of color and label are less transparent. What are those objects? This means those columns can contain any type of the object. At the moment, we have strings in them, but what we really want them to be are categorical variables. In case you don't remember from the previous chapter, categorical variables are like Swift enums. Fortunately for us, data frame has handy methods for converting columns from one type...

Decision trees everywhere

The algorithm that we're going to use for our first machine learning exercise is called a decision tree classifier. A decision tree is a set of rules that describe the process of decision making (see figure 2.5 for example).

Decision trees are widely used outside the machine learning in different domains; for example, in business analysis. The popularity of decision trees is understandable: they are easy to interpret, and nice to visualize. For many years, they were built manually using the domain expert knowledge. Fortunately, now we have machine learning algorithms that can easily turn almost any labeled dataset into a decision tree.

Training the decision tree classifier

Let's learn how to train the decision tree classifier as shown in the following code snippet:

In []: 
from sklearn import tree 
tree_model = tree.DecisionTreeClassifier(criterion='entropy', random_state=42) 
tree_model = tree_model.fit(X_train, y_train) 
tree_model 
Out[]: 
DecisionTreeClassifier(class_weight=None,  
            criterion='entropy', max_depth=None, 
            max_features=None, max_leaf_nodes=None, 
            min_impurity_split=1e-07, min_samples_leaf=1, 
            min_samples_split=2, min_weight_fraction_leaf=0.0, 
            presort=False, random_state=42, splitter='best') 

The most interesting for us are the class attributes of DecisionTreeClassifier:

• criterion: The way to estimate the best partition (see the How decision tree learning works section).
• max_depth: Maximum tree depth.
• max_features: The maximum number of attributes to account in one split.
• min_samples_leaf: The minimum number of objects in the leaf; for example, if it...

How decision tree learning works

Decision tree learning is a supervised, non-parametric algorithm used for classification and regression.

Building a tree automatically from data

The Twenty Questions game is a traditional game where one of the players is the answerer who chooses an object (or a famous person in some variants), not revealing what it is to the other participants. All the other players are trying to guess what the object is by asking questions like Can I eat this? or Is it a human? where answers can only be yes or no.

If you have never heard about this game, refer to Wikipedia: https://en.wikipedia.org/wiki/Twenty_Questions.

This is essentially a tree learning algorithm. To win in a game, you should pose such questions that discriminate the most; for example, the question, Is it alive? in the beginning of the game is clearly better than Is it a cucumber?. This ability to dissect the hypothesis space in an optimal way is formalized in the notion of information gain criterion.

Combinatorial...

Implementing first machine learning app in Swift

You can transfer your model from Python to Swift in two ways: transfer a trained model, or train a model from the ground up in Swift. The first option is easy in the case of decision trees, as a trained model can be expressed as a set of if-else conditions, which is trivial to code manually. Training the model from the ground up is required only in the situation where you want your app to learn in runtime. We will stick to the first approach in this example, but instead of coding rules manually, we will export the scikit-learn model for iOS using Core ML tools.

Introducing Core ML

Core ML was first presented at Apple WWDC 2017. Defining Core ML as machine learning framework is not fair, because it lacks learning capabilities; it's rather a set of conversion scripts to plug the pre-trained model into your Apple applications. Still, it is an easy way for newcomers to start running their first models on iOS.

Core ML features

Here is a list of Core ML features:

• coremltools Python package includes several converters for popular machine learning frameworks: scikit-learn, Keras, Caffe, LIBSVM, and XGBoost.
• Core ML framework allows running inference (making predictions) on a device. Scikit-learn converter also supports some data transformation and model pipelining.
• Hardware acceleration (Accelerate framework and Metal under the hood).
• Supports iOS, macOS, tvOS, and watchOS.
• Automatic code generation for OOP-style interoperability with Swift.

The biggest Core ML limitation is that it doesn't support models training.

Exporting the model for iOS

In our Jupyter notebook...

Summary

In this chapter, we had our first experience of building a machine learning application, starting from the data and all the way over to the working iOS application. We went through several phases in this chapter:

• Exploratory data analysis using Jupyter, pandas, and Matplotlib
• Data preparation—splitting, and handling categorical variables
• Model prototyping using scikit-learn
• Model tuning and evaluation
• Porting prototype for the mobile platform using Core ML
• Model validation on a mobile device

There are several machine learning topics that we've learned about in this chapter: model parameters vs. hyperparameters, overfitting vs. underfitting, evaluation metrics: cross-validation, accuracy, precision, recall, and F1-score. These are the basic things that will be recurring topics throughout this book.

We've become acquainted with two machine learning algorithms, namely decision trees and random forest, a type of model ensemble.

In the next chapter, we're going to continue exploring classification...